0:09

Hello and welcome back to Talk Evidence.

0:11

It's Christmas again and we've got a very

0:13

special podcast for you today with a

0:15

very special guest. It's not

0:18

Father Christmas himself. It is

0:21

Tim Feeney, who is joining us from

0:23

the BMJ research team who has

0:25

been heading the selection of

0:28

Christmas papers for the very special

0:30

issue that comes out once a

0:32

year. Hi, Tim. Hi,

0:34

everyone. Thanks for having me. We're

0:36

also joined as usual, Biopharm Franco,

0:39

editor and chief of BMJ, BMJ,

0:41

and also a GP. Hi, everyone.

0:44

And myself, As always, Helen

0:46

McDonald, I'm publication and content

0:49

integrity editor across B and J journals

0:51

and clinical editor on the B and J.

1:03

We've got a really interesting selection of papers

1:05

for you this year, and I feel like it's kind of

1:07

reflecting some themes that we've seen through

1:09

twenty twenty. So we're going to begin with

1:11

a topical look at some news on the house of footballers

1:14

as the World Cup has just come

1:16

to an end. We're gonna look at the

1:18

performance of BMJ editors

1:20

at picking papers that are highly cited

1:23

or perhaps not. And we're also going to look at

1:25

the performance of AI, artificial

1:27

intelligence machines versus radiology

1:29

exam candidates We're going to

1:32

zoom to misinformation and a belief

1:34

that everything causes cancer. And

1:36

finally, some tips from BMJ statisticians

1:39

to set the world right before we start twenty

1:42

twenty three. Tim,

1:52

I'm gonna hand over to you to tell us a little bit

1:54

about this paper on elite footballers.

1:57

And alcohol consumption to

1:59

topical themes? Yes,

2:02

we thought that this was quite

2:04

a quirky question that would be a ventures

2:06

particularly this year since the

2:08

World Cup is happening. As

2:11

we speak, I was watching it last night.

2:13

And so in this study, they basically

2:15

wanted to see if elite footballers in

2:18

Sweden, in the Swedish top tier

2:20

footballers which akin to the

2:22

premier league. And they looked at footballers

2:24

to determine if they were at a higher risk

2:26

of alcohol related disorders. And

2:29

they found that, overall, they were not

2:32

despite the,

2:33

perhaps, sometimes you feel that footballers

2:35

may be more app to

2:38

drinking party. And

2:39

then interestingly, when they looked at

2:42

age and year

2:44

of starting in the top tier,

2:46

they found that more recent

2:49

footballers were actually had a lower

2:51

hazard of having

2:53

alcohol related disorders. But

2:56

that older footballers, people

2:58

that were previously players, maybe

3:00

had a higher risk. So which you

3:02

can kind of piece together at least what what

3:04

we thought we saw on this was that

3:07

maybe a long time ago, footballers

3:10

may have been slightly increased

3:12

risk or had a slightly higher

3:14

hazard of alcohol related disorders.

3:16

But more recently, maybe because

3:19

of scrutiny or societal changes that actually

3:21

that going down. And so they're maybe

3:24

more focused on healthy behaviors

3:26

that allow them to perform at the top level. We

3:29

thought it was interesting. It's interesting

3:31

because maybe people don't necessarily

3:34

when they think of football, think of the

3:36

elite athletes that play football

3:38

in Sweden. But obviously,

3:41

as nerds, we all know

3:43

that they have amazing healthcare

3:45

records and databases of all

3:47

kinds in Sweden. So this is

3:50

tremendous study. Tim, you were talking about

3:52

the you know, differences over time.

3:54

This is hugely long, isn't it? Tell us

3:56

when it started and when it ended. I mean, it's just

3:58

massive. Sort of like a whole

4:00

generate multiple generations of

4:02

footballers. Yeah. The data spans

4:05

from nineteen twenty four to twenty twenty.

4:07

So it's a it's a long it's

4:09

a long period of time that we were able to

4:11

evaluate this. And

4:14

I think that you're right, that really does

4:17

that really is what strengthens the study is that

4:20

even though as you mentioned you wouldn't think of Sweden

4:22

when you think about football, you do think of

4:24

Sweden when you think about excellent

4:26

link data. Excellent link data.

4:28

That that's what a lot of our listeners, I think, are

4:30

probably thinking as well. Yeah. So

4:35

I don't know. It depends on what circles you roll

4:37

in. Exactly. I mean, we we

4:39

have very cool circles with MJ.

4:41

It's just been asked Satisticians meeting

4:43

and our research editor's meeting, and I

4:45

love seeing you want for

4:47

humans. List

4:50

stick. Sorry. Let's stick on

4:52

the topic of time because while another

4:54

of the interesting things that you picked out about

4:56

this paper was not just looking

4:58

at this over multiple generations,

5:00

but actually looking through the duration

5:02

of the footballers lives. So both the

5:04

time that they were playing football, but then

5:06

after they'd retired, and how things changed

5:08

over that time?

5:13

Yes, because when people

5:16

think about food bars, especially when

5:18

they did this analysis about how many

5:20

scores they they got and

5:23

successful they were and they were

5:25

unable, perhaps, underpowered detect

5:28

whether there was a higher risk for

5:30

this subpopulation. But people tend to think

5:32

that perhaps when the football rollers

5:34

retire, their

5:36

40s or perhaps late 30s,

5:39

their life change – their

5:42

life changes and and

5:44

perhaps that may lead to a higher

5:47

risk of substance

5:49

abuse disorder. There are famous cases,

5:51

for example, my donna from Argentina that

5:53

everyone knows that he suffered from

5:55

several addictions and died

5:58

in twenty twenty. And

6:01

those cases are known widely

6:04

enough for the public, and they may think

6:06

that it might be

6:08

generalizable to other football

6:10

players. But this study

6:12

shows that perhaps the healthy

6:14

habits of of following the sports may

6:17

leave on. So

6:28

from elite athletes to elite

6:31

editors, And one, you're gonna

6:33

tell us a little bit more about this paper

6:35

on whether the j's

6:38

research editors are able to

6:40

predict which research articles

6:42

will be highly cited. Tell

6:44

us about it. So,

6:47

yeah, this this was the research that

6:49

was bake at home at the BMJ. So

6:51

Eight seven. Like a big pie.

6:54

A group of managers to the research

6:56

papers from twenty fifteen, twenty sixteen,

6:59

and including those accepted and

7:01

rejected from VMJ, and

7:03

they analyzed their their

7:07

what were they feeling? Their feelings

7:09

about them in their manuscript meetings. So

7:11

the BMJ editors need weekly

7:13

and discuss the papers and then

7:15

decide whether to accept or reject. But

7:17

at that stage, sometimes we discuss

7:20

about our enthusiasm on how

7:22

well the uptake of the article would be.

7:24

Right? If they're gonna be changing

7:26

practice, if they're gonna excited

7:28

if they're gonna influence guidelines. And

7:32

and the and this paper looked followed

7:35

follow-up of the data on this manuscript.

7:38

Whether accepted or rejected. And

7:40

what they found is that nine out

7:42

of ten editors were unable to identify

7:44

the correct citation category of

7:46

paper. That means that most

7:48

editors said the papers were

7:50

going to be less sited than they were

7:53

actually were. So In

7:56

general, there there was this misclassifications and

7:58

editors were not able to predict

8:00

which articles were gonna be, like,

8:02

the big hit. So we're basically pessimistic.

8:06

Exactly. That's And I think researchers would

8:08

probably tell us that wouldn't they? They they would

8:10

recently underline the importance of their work,

8:12

and we continue to reject their work.

8:15

Well, the rejection letter says, and BMJ

8:17

accepts less than five percent of all submitted

8:19

manuscripts. It's it's

8:21

it's quite a tough. Tim,

8:23

you've been very quiet. And in fact, you

8:25

are one of these editors. Although, we should

8:27

say you were not one of the ones that were studied because you

8:29

weren't here in twenty, twenty fifteen,

8:31

twenty sixteen when these data we gathered,

8:33

but but give us your perspective on it.

8:35

Yeah. I like to think that I would have been an

8:38

outlier, but maybe That

8:39

one that one added the ten that actually

8:42

was good Yeah. I think,

8:44

you know, I find this it's

8:46

a fun read, and it's as

8:48

an

8:48

editor, it's fun to

8:51

get a self evaluation about

8:54

how we are at this. But

8:56

I will say that since

8:59

before joining as an editor, thought

9:02

that impact factor

9:04

probably was something that weighed on the minds

9:06

of research editors. But actually,

9:08

since I've joined, it's not something we

9:10

really take into account. So it doesn't really

9:12

bother me that we weren't able to predict

9:14

citations because it's not something

9:16

that really weighs on us as we're

9:18

trying to decide papers to take.

9:20

We think about how rigorous it is and

9:22

whether or not it's adding

9:24

something to the literature

9:27

And whether or not we think it's actually going to help

9:29

clinicians, we have a specific goal

9:32

in mind, particularly at the DMJ.

9:34

And, you know, citations are

9:37

one aspect of

9:39

amiriata things we're concerned about.

9:41

So we were talking earlier and, you

9:43

know, would have

9:44

a paper is not cited a lot, but physicians

9:47

still find it useful, then it still made an

9:49

impact, but that's not measured anyway.

9:51

And that's an interesting point, isn't it?

9:53

Because there's so many different ways you could

9:55

measure the impact of a paper.

9:57

But actually, the impact factor is

9:59

very measurable whereas some of those other

10:01

measures about whether it changed

10:03

practice, whether that's your personal practice,

10:05

or whether it changed policy and

10:07

influence things by guidelines,

10:10

a lot of that impact is very

10:12

hard to measure. And it's

10:14

it will be interesting to know if

10:17

the research editors are at any

10:19

better predicting those things. If we ever end up

10:21

with good ways to measure

10:23

broader impact? Yeah.

10:26

It's true. There's many ways to

10:29

measure impact and citations

10:31

alone don't just don't do

10:32

it. So

10:32

we try not to let

10:35

that creep in into our thought process

10:37

at all, which, you know, so

10:40

ultimately, these results don't

10:42

worry me that we're not doing a good job.

10:44

Essentially, it's it's more fun to

10:46

just look at it and say, oh, yeah. We we

10:48

really can't really peg it, so

10:50

we might as well not even try. And

10:52

I think some of our listeners will enjoy listening

10:54

to this. They can poke some fun at

10:57

BMJ's research editors and

10:59

know that Maybe in some ways we're

11:01

not gonna add germs. We're

11:10

going to stick with performance related

11:13

issues. We've talked about the

11:15

performance of editors, let's talk

11:17

about the performance of artificial

11:19

intelligence. So at

11:21

least in the UK, we've heard a lot this

11:23

year about the big shortage of

11:25

doctors that we have And with that mind, perhaps

11:27

it would be good if some of the roles that

11:29

doctors do could be

11:31

enhanced or replaced by

11:34

automation. Is it that much

11:36

of a jump to ask whether

11:38

AI systems could be as good as

11:40

radiologists or at least ask whether they

11:42

can pass UK radiology

11:44

exams. Tim, would you tell us a bit more about

11:47

this paper? Yeah,

11:49

I think it's really topical now, especially

11:51

with all the hubhub about chat

11:53

GPT. But

11:55

basically, this was a study looking

11:57

at a

11:59

very frequently used

12:01

artificial intelligence radiology

12:06

application that read radiographs

12:09

and compared it to

12:12

physicians that just passed

12:14

the certification

12:16

exam per radiologist and basically

12:18

compared the scores. And they

12:20

found that the AI radiologist even

12:22

in the case of radiographs

12:26

that the AI was trained on is not

12:28

quite as good as a human radiologist

12:31

indicating that there

12:34

still needs to be a lot of improvement for

12:36

an AI before it can take over for a

12:38

position. And maybe it's

12:40

it's optimal role right now is

12:42

augmenting radiologists to help them

12:44

find

12:46

more subtle findings. Or maybe at

12:48

least raise the profile of some patients to get

12:50

a deeper

12:50

look. And we thought this

12:53

was very topical

12:56

given the concern about AI and where what

12:58

its role in medicine really

13:00

is. Yeah.

13:02

And BMJ has published a few pay papers on

13:04

this. I remember, Tim, I

13:07

think we published a systematic review

13:09

not that long ago looking at AI

13:11

studies in a different field,

13:14

which showed that they

13:16

really weren't quite there yet. So

13:18

interesting to see how

13:20

the field's

13:20

progressing? Anne, what were your thoughts on this

13:22

one? Well,

13:24

it's interesting to look at the

13:26

overall picture of

13:28

this evaluation. But

13:31

I found it interested when

13:33

that the performance was quite

13:35

different and quite limited the way we

13:37

can compare performance of

13:39

AI and radiologists when you break it

13:41

out by diagnosis. Mhmm. And

13:43

we know that not all

13:45

radiological diagnosis are similar. And there

13:47

are, for example, the fracture of the

13:49

distal filings of a big toe.

13:51

It's just a completely different

13:53

diagnosis from. From

13:55

a from a

13:57

female fracture. And

13:59

and I think that we cannot expect

14:01

the the performance of AI

14:03

to be the same across all

14:05

diagnoses or to have the

14:07

same role. And did it show you a hope

14:09

in some areas? Did you find

14:11

a Yeah.

14:13

So for the May some of the main areas in

14:15

which it had better performance

14:17

has to do with pulmonary and

14:19

thoracic diagnosis, which

14:22

is sometimes like, the the most

14:24

obvious diagnosis in itself. So

14:26

let me let you say that as well. If

14:28

you if you it's if if you

14:30

on the on the contrary, if you look at one of

14:32

the pictures that shows

14:35

a misclassification by AI,

14:37

it says, normal lateral

14:39

scapular wide view of

14:41

a shoulder of a child was

14:43

misinterpreted as a a

14:45

humeral fracture. So that tells

14:47

you that. And and and everyone has done

14:50

we've seen radiographer from children's

14:52

or know that there there are

14:54

quite challenges sometimes to

14:55

read. So is that the most

14:58

humorous aspect of this paper for you?

15:00

I I

15:03

almost said Maybe Helen has a bone

15:05

to pick with the AI. Okay.

15:23

We are crashing out of that

15:25

hilarity from Tim into

15:28

something a bit more serious. To

15:31

this paper. I restored to

15:33

this paper. Everything

15:36

causes cancer was its title.

15:39

And I was sure to it because it's it's the kind

15:41

of thing which I kind of

15:43

believe at some level

15:45

probably because I've got too little filter

15:47

between my thoughts and what comes out my mouth. And it's

15:49

not very christmasy or funny, but it makes

15:51

quite a clever point about

15:54

misconceptions and misinformation. And

15:56

I think that has been such a pervasive

15:58

problem across society for the

16:00

last couple of years that

16:03

this this is worth Sorry.

16:05

I'll just rephrase that last bit.

16:10

This feels like one of those Christmas

16:12

papers that is

16:14

quite novel or seems quite

16:16

quirky, but actually gets at something

16:18

which is quite serious.

16:20

So it's a cross sectional study,

16:23

which we've conducted on a variety of

16:25

online platforms. Some of them

16:27

listeners might recognize one of them is called Reddit,

16:30

which upgrades content

16:32

depending on its use, and there are a few others

16:34

in Spanish language. And

16:36

this is the kind of content that gets fed to you

16:38

on your social media feed. So if like

16:40

me, sometimes you sit crawling and you

16:42

see these kind of bizarre articles

16:44

that are served to I guess, serve quite

16:46

a lot that are about aliens. I don't know what

16:49

I must do to

16:51

get that content and Tim and Harla

16:53

NICE and Coir. Obviously, they don't get sublogy

16:56

content on their social media feed. Just

16:58

makes me wonder what else you're

17:00

talking about? Because the AI

17:02

is survey there. Yes. Anyway,

17:06

these authors designed to pop up

17:08

survey to appear on these sites. And they

17:10

asked people who were who were

17:12

viewing this content about their health beliefs.

17:14

So they'd asked them about their

17:16

thoughts about conventional and alternative medicines

17:19

about whether they had had COVID-nineteen vaccinations

17:21

and their reasons for not undergoing

17:23

those. They asked them about conspiracy

17:25

theories, some of which I've never heard of,

17:27

so I'm obviously not the target

17:29

audience for some of these

17:32

around things like the world being

17:34

flat and reptilians. I'd

17:36

never heard of that one. And then they

17:38

ask about about cancer.

17:40

So they ask about risk

17:43

factors for cancers. So they look

17:45

at these eleven established things,

17:47

which are linked cancer. So

17:51

smoking, alcohol, low levels of

17:53

physical activity, eating red

17:55

and processed meat, sunburn as a child

17:57

family history of cancer, these kind of

17:59

established factors.

18:01

And they also ask them questions on

18:03

risk perception, which which

18:06

are not established

18:08

causes of cancer, and

18:10

they classify these as kind

18:12

of mythical, although I

18:14

think more accurately than non established causes than

18:18

necessarily totally fictitious. And

18:20

these include things like

18:22

drinking plastic from bottles, eating food

18:25

containing artificial sweeteners, genetically

18:27

modified food, use of microwaves,

18:30

aerosols, mobile phones, cleaning products,

18:32

this type of thing. And then

18:34

they ask the participants to agree

18:36

with a certain number of things. And what

18:38

they hone in on discussing

18:40

some of these main results is this statement

18:42

that almost half of the participants

18:45

agreed with the statement that it seems like

18:47

everything causes cancer. Just kind of how I feel

18:49

a bit of the time when you

18:51

read at least a lot of health

18:54

research papers. And

18:56

it kind of does make a serious point

18:59

that out there in the public, there is

19:01

difficulty understanding what causes

19:03

cancer and that actual

19:05

awareness of either

19:09

actual or mythical or

19:11

unconfirmed cancer causes is

19:13

quite low. And they feel the authors that

19:15

this suggest a connection

19:17

between digital misinformation and

19:19

health decisions. And they

19:21

and they suggest that

19:24

education and better scientific

19:26

literacy is needed to

19:29

help build and better health

19:33

communication. So I I quite like this

19:35

paper. It's kind of it grows on you

19:37

slowly. It's not this

19:39

virus is some of the

19:40

others. Tim, what did you think of

19:43

it? Well, I think this is a good

19:45

opportunity for me to reiterate

19:47

that Christmas papers are not necessarily only

19:50

interesting when they're funny. I feel like a

19:52

lot of authors feel like that's the the main

19:55

the gist. And while we enjoy a humorous

19:58

paper, this is an

20:00

example where making

20:02

another point and having a quirky question

20:04

is, the overriding objective

20:07

here. And so I think we

20:09

were most interested in this as it

20:11

pertains in this information particularly

20:13

because we live in

20:15

a time where we're still

20:18

in the pandemic, and there's still

20:20

a lot of questioning about the interventions

20:22

that work don't work in terms

20:24

of that. And we found that

20:26

both it was interesting

20:29

that those that had spiracy

20:32

beliefs were more likely to

20:34

believe in the non

20:36

established cause of cancer, but we

20:38

also found it worrisome that so

20:40

many people kinda

20:42

have misinformation overload.

20:45

And they just kinda put their hands up and say, well,

20:47

I guess everything causes cancer. And so

20:49

some of those mythical reasons causes, you

20:52

can understand why you may think that they

20:54

have caused cancer and there's just

20:56

no evidence to support that they

20:58

do yet. But I think that shows

21:01

that people in

21:04

the entire community

21:06

of science communicators and

21:08

health communicators need to be judicious

21:10

in what they inform

21:13

the public with so that they don't

21:15

become overloaded and that they can

21:17

really focus on addressing the causes

21:20

of cancer that we know

21:22

to be established and can be

21:24

prevented. We don't want it to get

21:26

lost and we don't want people to kind of throw their

21:28

hands up. And say, well, I guess I can't do

21:30

anything. So I might as well not try and prevent

21:32

cancer because there are things you should you shouldn't

21:34

smoke, you should limit the amount of

21:36

alcohol you consume, you

21:39

should be active in trying to maintain a

21:41

normal body weight. You should use

21:43

sunscreen when you're out in the sun. Those are all things

21:45

we know you should do. So we don't want I

21:47

think it would be a detriment to

21:50

society if we confuse people

21:52

so much that they can't discern what what the

21:54

most important things are to take care of their

21:56

health are. I'm aware

21:58

of those.

22:00

Well, perhaps I'm I

22:04

was a little bit less concerned because

22:07

we have to acknowledge how much

22:09

of our lives we live with

22:15

wrong bill perhaps

22:17

deceptive beliefs of our our reality.

22:20

And at the same time, what's the role

22:22

of the scientific community

22:24

in feeding into this this information?

22:28

Every every other year, get a paper that

22:30

says that either

22:32

coffee causes cancer or coffee

22:34

prevents cancer. So and

22:36

all signal goes into the population in different

22:39

ways and they sit and they

22:41

sit on subgroups of people

22:43

differently. Of course, if you're a couple of you're

22:45

gonna love your and the papers

22:47

that said that it prevents cancer. But

22:49

if it causes insomnia,

22:51

then you said, oh, I can't sleep when I drink coffee

22:53

and besides it causes cancer. So

22:55

a lot about the attitudes that needs

22:57

to be modulating against. And if if

22:59

you think about it, the the

23:01

people who believed in reptilians still

23:04

believe that smoking cause cause cancer. And

23:06

that's reassuring. Perhaps

23:08

they're not gonna stir their food in the

23:10

microwave, while they're cooking,

23:14

but III consider that

23:16

that like a let's have two evils.

23:18

So so I think we

23:20

should work on the attitude, basically.

23:31

This will

23:39

be our last episode of Talk Evidence

23:42

before the New Year. So for any of you looking

23:44

for some New Year's resolutions,

23:46

our final paper from the pick

23:48

of BMJ's Christmas content is

23:51

one from our statistician to have

23:53

some pointers for you about how to

23:55

conduct and report your work, and we should

23:57

feel very privileged to have this because

23:59

quite often we ask them for things

24:01

like tell us how to do things write down

24:03

all of your wisdom so that we can follow it. And they're always

24:05

very good them do it.

24:07

So it's very exciting to see that they

24:09

finally decided to gift

24:11

us with all of their wisdom for

24:13

Christmas. It's a play. It's a

24:15

play on the song in the twelve days of Christmas,

24:17

and it's authored by statisticians.

24:19

Are there twelve of them, Tim?

24:21

Do you think they each got one in here?

24:27

Yeah. Some of them more than

24:29

others maybe. Statisticians,

24:32

I love statisticians. I wish I was as

24:34

careful and as thoughtful as as statisticians. And

24:36

I I loved reading some of their phrasing,

24:38

introducing this paper. They say,

24:40

A small but

24:43

influential group of individuals

24:45

with a very shiny nose for

24:47

detail. That's them, isn't it? That's

24:49

them. Yeah. Do you have a better sense of

24:51

humor than I thought? I know.

24:53

Seeking all his right

24:55

rather than all his right

24:57

and emphasizing the no no

25:00

no no rather than the ho ho

25:02

ho. We call ourselves

25:05

statisticians and our core belief is

25:07

that research articles are for

25:07

life, not just for Christmas. They put

25:10

a lot of thought into those words

25:12

for statisticians, I think. I

25:14

think they did. So

25:17

they did make twelve wishes. I don't think we

25:19

have time to share their twelve wishes

25:21

with you, but I think we can maybe

25:23

share our highlights from

25:26

them. So, Juan, having read their

25:28

very careful instructions

25:31

to improve your research going forward.

25:33

What what did you take home?

25:35

Well, I like to

25:38

pick up two. One

25:40

is number four. Yeah.

25:42

I'm seeing it.

25:43

That's all day of

25:46

Christmas. No. Do

25:48

not dichotomize continuous variables.

25:50

That would

25:51

make it great too. Yeah.

25:54

Yeah. Centralize dichotomization. You are

25:56

either either naughty or nice, but you will pop

25:59

Paul if you choose to adopt a miscellaneous

26:01

variables. And this is something that you hear

26:03

a lot from the set of

26:05

decisions because there are there are many arbitrary

26:07

decisions as to when you the customer is Juan,

26:09

just give us an example.

26:12

Well, for example, if you're doing a logistic regression

26:14

and you use age as a covariate, and

26:18

you said, well, people who are

26:20

a certain age over fifty

26:22

years old or below six years

26:24

old and use

26:26

that to build a model, then some people

26:28

say, why didn't you use fifty nine or

26:30

sixty one as they cut point.

26:33

And and it creates the idea that

26:35

those two categories are very different, but one

26:37

of the categories could include people in

26:39

in their and or in

26:41

sixty one and the other category Grain

26:43

group, we were in the twenties. And so you lose

26:45

a lot of data, which is another one of the

26:47

points of what why it

26:49

says that uses. It's a it's

26:51

a difficult power. I'm sure the suggestion

26:53

will sprint this better than it, but

26:55

hope that you get the gist. So

26:57

it it is something that comes frequently

26:59

when we asked for stats report on

27:02

on research. Okay.

27:04

Tim, give us your give us your

27:06

first one.

27:06

I'm gonna say,

27:09

I really enjoyed the

27:11

part about your study question.

27:14

And having a well formed study

27:16

question. The fundamentals Yeah.

27:19

Oh, yeah. This is not just something that the

27:21

statisticians worry about, some of the

27:23

research editors constantly grapple

27:25

with determining what the question

27:27

is, having the

27:29

question sharpened to the point where you can

27:31

answer it sufficiently, And

27:33

then being consistent with that question, like,

27:35

doing analyses that try to

27:37

answer the question, not beat

27:40

around the bush. So that's where I'm gonna

27:42

stick my first flag.

27:44

I like that. I I'm going

27:46

niche on my first one, actually.

27:48

I'm going for subgroups.

27:52

On the sixth day of Christmas, quantified

27:55

differences in subgroup results.

27:57

So they say here that a common mistake

28:00

is to conclude that the results from

28:02

one subgroup are different to the results

28:04

from another subgroup without actually

28:06

trying to quantify the

28:08

difference between those results.

28:10

And that might lead you

28:12

to naively conclude that

28:15

a treatment is beneficial for one subgroup

28:17

of patients, but not for another.

28:20

However, the statisticians say that actually

28:22

when you compare the results between

28:24

those subgroups, you often find that

28:26

there are quite wide confidence

28:29

intervals. Which usually suggests that

28:31

further research is needed before

28:33

concluding any subgroup effect. So

28:35

that was a very serious message. Let's

28:37

come back to quite if you got a serious or

28:39

a silly Christmas message for your next one.

28:43

Yeah. Sort of serious, I would say.

28:45

The number eight interpret I squared

28:48

and mental regression appropriately.

28:50

That sounds very serious. Say it again slowly.

28:54

Impet I squared, and

28:56

better regression appropriate. Better

28:58

regression. Metoid regression.

29:00

Metoid regression. Metoid regression. Sorry.

29:02

And so the basically, there's

29:05

a definition about I

29:07

squared. So when you're doing a meta

29:09

analysis, when the missions heterogen

29:12

a d is I squared and a

29:14

d is usually misinterpret and I

29:16

want to read it from here it

29:18

says I squared describes a percentage of

29:21

variability in the treatment

29:23

effects estimates that is due to

29:25

between study heterogeneity rather

29:28

than chance So basically, usually researchers

29:30

try to think about, oh,

29:32

how variable is the result across study?

29:34

And that's like of

29:36

raw interpretation of I squared, but

29:39

actually is from the estimate of the

29:41

mechanism, how much is

29:43

between because of the between

29:46

study differences? So it

29:48

actually tells you

29:51

how much of the

29:53

variability across studies affecting the results of the

29:55

meta analysis in a way, which I hope

29:57

is not a misappreciation. I will

30:00

classify them at this point. But

30:02

and I really like that because

30:04

Richard Reilly, which he probably

30:06

influenced this sentence. He

30:08

tweets once every month

30:10

this definition is to their profile.

30:13

Know It's just exactly the same tweet. It just

30:16

keeps retweeting every month. So I tried to

30:18

put a like and retweet when they didn't see

30:20

that. No. Well, I

30:20

think we're she's gonna like you today, hon, for

30:23

mentioning that one. What what else

30:25

have you got first in? Oh,

30:28

number two. Yeah, I picked the first I

30:30

guess. Focus on estimates,

30:32

confidence, intervals, and clinical relevance. That

30:37

that doesn't fit with two turtle doves

30:39

because that's the line from the song, isn't it? Number two

30:41

is two turtle doves. That's that's a lot

30:43

of this.

30:44

Very big present. Tell us

30:46

again? Yeah. So I

30:49

think it's really important that

30:53

in this day in

30:55

Egypt authors move away from Noel

30:57

Hypothesis Significant Testing

30:59

and moved more toward

31:02

estimating effect sizes,

31:06

estimating confidence intervals for

31:08

those effects, And then

31:10

determining what the clinical relevance of those

31:12

effects

31:12

are. I think that's the most important thing.

31:14

That's the thing that will last

31:17

It's very hard to interpret something

31:20

when you just determine that they're

31:23

different enough

31:25

and saying things like trending towards

31:28

significance is just the

31:30

worst. So

31:30

I I am very

31:33

much on board with the

31:35

determining your good your best question, what you're

31:37

trying to estimate, and then estimating

31:41

and getting confidence intervals. So I

31:43

feel like I overcomplicated it with this my

31:45

subgroup analysis. One, so

31:48

I'm I'm returning to safe ground

31:50

for me. For for

31:52

my final choice. And

31:54

actually, this was the final the

31:56

final one of the song. On the

31:58

twelfth day Christmas, the BMJ statistician

32:00

sent to me, advice, to

32:03

use reporting guidelines and avoid

32:06

over interpretation. Hold

32:11

on. And

32:17

they include a quotation from

32:19

Doug Altman, late Doug Altman

32:21

who was BMJ's

32:23

chief statistician for a long time, which

32:26

says readers should not have to

32:28

infer what was probably done.

32:30

They should be told explicitly. Proppant

32:33

methodology should be used and seemed

32:35

to have been used.

32:39

So this point is about making use

32:41

of reporting guidelines, making use

32:43

of the checklist of items

32:45

that they tell you you need to

32:48

report so that research is

32:50

communicated clearly and

32:52

that its results can

32:54

be clearly put in context.

32:56

So they're not spun, they're not over

32:59

interpreted, and everyone

33:01

knows what you're talking about.

33:06

If that

33:13

feels like it's a bit of

33:15

a serious issue to finish on because it is something that I feel quite

33:17

passionately about. You could revisit some of

33:19

our Christmas content from a different year

33:22

and Tim pointed out had a very funny paper last

33:24

year from a researcher who wrote a

33:26

fake reporting guideline called

33:29

bogus it's sort of an anti reporting

33:31

guideline tells you all the things that you

33:33

shouldn't do and is probably very contrary to

33:35

everything that's contained in the twelve days of Christmas

33:37

from our statisticians. So

33:40

that's all folks. Goodbye

33:43

to twenty twenty two.

33:45

Thank you to Tim for joining us.

33:47

We'll see you next year, perhaps,

33:49

for the twenty twenty three installments

33:51

of Christmas papers. And Juan, I'll

33:53

see you in a few weeks with

33:56

Joe. To get back to our regular recording of talk evidence

33:58

where we'll be discussing the best new

34:01

evidence and perhaps some more of your

34:03

evidence challenges. Faced

34:05

by our listeners. Until then, it's

34:07

goodbye for me. Goodbye for me.

34:09

Bye for me too. Thanks

34:11

for having me. So merry Christmas or

34:13

happy holidays, and we'll see you in

34:15

the New Year.