0:05

Hello and welcome back to Talk Evidence,

0:07

your monthly look at the world of evidence from

0:10

BMJ's editors. I'm Helen MacDonald,

0:12

Content Integrity and Publication Ethics

0:14

Editor at BMJ and I'm joined on the podcast

0:17

by Juan Franco, Editor-in-Chief of

0:19

BMJ-EBM, evidence-based medicine,

0:22

GP and systematic review fanatic.

0:24

Hi Helen, hi Joe. We're also

0:26

joined by Joe Ross, the BMJ's US Research

0:29

Editor, General Internist and Big

0:31

Data fan.

0:32

Hey everybody, nice to see you. This

0:39

week we've got lots on the menu and we're talking

0:41

about pay for performance measures.

0:44

A research paper from Scotland tracked what

0:46

happened when some quality outcome

0:49

framework indicators or QOF as it was

0:51

known for GPs were stopped. We'll

0:53

talk about generating data for

0:55

research and quality improvement in primary care.

0:59

We've got a special interview with Hussein Najoo

1:01

from the London School of Economics about the

1:03

UK government's new plans for drug

1:05

regulation after Brexit. We'll

1:08

touch on the US spend on mRNA

1:10

vaccines and finally Juan is going to shed

1:12

some light on how good or not point-of-care

1:15

medical apps are for doctors. Let's

1:23

get started with QOF, the pay

1:26

for performance measures and Joe you're going

1:28

to tell us a bit about this paper.

1:30

And I had no idea it was called QOF. Oh

1:33

it is, it's much easier to say than pay for

1:35

performance measures which I keep tripping over.

1:38

It's so funny because in the United States

1:40

what we would say is P4P has been

1:43

around for decades upon decades

1:45

as a way to improve quality

1:47

because right in the you know the idea

1:50

being like if we're going to

1:52

find ways to improve quality

1:54

and performance what's the surest way to do

1:56

it but let's pay for it. And so this

1:59

idea of

1:59

of trying to get physicians

2:02

and hospitals to provide

2:04

the most appropriate highest quality care

2:07

using financial incentives, has

2:09

been around for a long time, but we saw an explosion

2:12

in these types of programs, not

2:14

just in QAF, throughout the UK,

2:16

but also throughout the Medicare

2:19

program in the United States and other countries throughout

2:21

the world, as we all became

2:23

aware of the problem of kind

2:25

of providing low quality or low value

2:27

care, and this being kind of

2:29

the sort of front and center kind of easiest

2:32

solution. What I liked about

2:34

this paper, which came from a great group

2:36

in Scotland, is they kind of, they asked the question

2:38

of, well, you know, there's

2:40

been kind of marginal studies that suggest

2:43

when you quote unquote pay for performance, you

2:45

get better performance, but there's always been

2:47

a lot of skepticism about,

2:50

does that actually improve outcomes for patients or

2:52

does that make physicians and clinicians

2:54

just do a better job of ticking the box, saying

2:56

that they provided mental health screening

2:59

or tobacco cessation counseling, but the actual

3:03

effectiveness of those interventions is questionable,

3:05

because they're just kind of ticking boxes. And there

3:07

are some measures that are looking at

3:10

actual kind of intermediate related

3:12

outcomes, like someone's hemoglobin A1C

3:14

measure or their blood pressure and

3:17

so on and so forth. But what was interesting about

3:19

this paper is it's evaluated

3:22

essentially what was a natural experiment,

3:25

where the QOF program had

3:27

grown to such a size across

3:29

the UK that Scotland essentially

3:32

decided to abandon it, I believe

3:35

in around 2016 or so, but

3:37

they continued to collect the data. So

3:39

they have this interesting kind of before, after

3:42

comparison, looking at the

3:45

measured performance in Scotland and

3:47

in England,

3:48

both before and after

3:51

only Scotland stopped providing

3:53

the financial incentives. And

3:56

I think what they find is kind of

3:59

in terms of like what the program

4:02

was doing. They have 16 different

4:04

measures of care, these sort

4:06

of performance indicators that were routinely

4:08

being measured. These are things like, did

4:11

you provide mental healthcare planning

4:13

and diabetic foot screening for patients with diabetes?

4:16

Do you, you know, are people's blood pressure

4:18

controlled if they have hypertension or diabetes,

4:20

or have had a stroke or peripheral arterial disease?

4:24

Was the hemoglobin A1C controlled among patients

4:26

with diabetes? And were

4:28

patients with

4:29

these certain comorbid

4:32

conditions like diabetes and COPD and

4:34

heart disease, did they get their influenza immunization?

4:37

And for patients with heart disease, did they

4:39

get their oral anticoagulation?

4:42

And they looked pre-post, you know,

4:44

kind of over time comparing to England.

4:47

And there's some really interesting figures

4:50

in the

4:51

manuscript where you can see

4:54

how essentially performance

4:56

drops, right? They stop clicking the

4:58

tick box for sure for

5:00

mental healthcare planning. It drops from like 60 some

5:03

odd percent down to below 40%. It

5:06

drops a little bit less for diabetic foot screening.

5:09

The blood pressure control as

5:11

having blood pressure less than 150 over 90

5:15

for many of these patients, all declines

5:17

a little bit, maybe like five to 10 percentage

5:20

points, which to me raises the question

5:22

of,

5:23

did people actually have worse blood

5:25

pressure control?

5:27

Or was the sort of gaming of

5:29

the measures eliminated

5:31

now that they were no longer being paid for it?

5:34

And you see other things, you know, you sort of,

5:37

you don't actually see as much of a drop in terms

5:39

of the treatment, the court,

5:41

like the getting the influenza vaccine

5:44

and getting the antiplatelet or anticoagulant. Those

5:46

basically stay about the same. So very

5:49

intriguing. I think the most important thing

5:51

is now gonna be a couple of years later, you

5:53

know, are patients, are their outcomes actually

5:56

worse? Or was

5:57

this just evidence of gamesmanship? But

5:59

it raises the question of, is a whole host of questions of,

6:02

you know, if we're going to try to improve quality

6:04

of care for patients and paying extra

6:06

for it doesn't work, what's the

6:08

best way to do it?

6:12

Wow, what were your thoughts? Well,

6:15

yeah, I was initially struck by the

6:17

choice of indicators. Some

6:20

of the indicators were

6:22

very complex and a lot of recording

6:25

and some of them also

6:28

related to some outcomes

6:32

for which we have been changing our

6:35

minds on how to think about it. For example,

6:37

in diabetes care, people who

6:39

are taking drugs to

6:42

reduce their glucose levels,

6:43

they might want to achieve a target

6:46

of HB1C and that

6:49

target needs to be customized to their own preferences

6:52

and needs and comorbidities. And

6:55

some of these indicators, I thought that

6:57

they were in light of 2023,

7:00

I guess, they seemed

7:02

a little bit less patient care,

7:05

patient centered. And

7:08

this sort of relates to this concept that

7:10

Victor Montorist says

7:13

about industrialized care and

7:15

how we're all sort of playing

7:18

by the system. And as Joe mentioned,

7:21

the distortions that might lead to

7:23

gaming of the system and

7:26

some of the concerns about whether

7:28

we're sort of playing a game

7:30

rather than improving care. Having

7:32

said that, I'm less

7:34

pessimistic about the withdrawal of

7:37

incentives because I'm not entirely

7:39

sure if you're a physician and

7:41

you've been working for these indicators for a long

7:43

time and you've been caring for patients and

7:45

sort of gotten used to working

7:48

in a certain way. If

7:50

you're withdrawn from these incentives,

7:54

are you going to do less

7:56

for your patients or perhaps are you

7:58

going to record less? of what

8:00

you do because you've taken this

8:03

incentive to tick all the boxes so

8:05

you look good.

8:08

At the end, these patients might have a

8:11

similar quality of care, but we're not

8:14

just saying that. I haven't

8:16

said that. There's an interesting analysis

8:18

piece that was published at the BMGA recently

8:21

that talked about the electronic

8:23

health records and

8:26

how the quality of data could

8:28

be improved. And they highlighted

8:30

this problem that the way we record

8:32

data in our systems

8:35

for research and for quality of care may not

8:37

be fully aligned

8:39

with improving patient

8:42

outcomes. And in this analysis

8:44

piece, perhaps we can link in the episode,

8:47

they mentioned some innovative solutions, for

8:50

example, natural language processing,

8:53

AI, or outsourcing

8:55

of data. So perhaps in the future,

8:57

what do they mean by that one? That those are

8:59

used during consultations to

9:02

fill in structured

9:05

data or how do they envisage them being

9:07

used? Well, basically,

9:10

in the electronic health care record, you have structured

9:12

data and unstructured data. So structured

9:16

data is the tick box sort

9:19

of thing, and the unstructured data

9:21

are the nodes. Yeah.

9:24

So the incentives work very

9:26

well when you're ticking boxes, but

9:29

a lot of the work we do sometimes is reflected

9:31

in the notes, which might under

9:33

represent what physicians actually do.

9:36

So if we actually can capture

9:38

what physicians do from the notes using

9:42

advanced tools on how to read those notes

9:45

with machines, and

9:48

perhaps we can capture those

9:51

actions from physicians that otherwise

9:53

would have been missed.

9:54

So it's about how you turn the

9:56

free text notes into more

9:59

structured data. Yes. And

10:03

they also mentioned outsourcing, but I'm guessing

10:05

Joe will probably have more idea on outsourcing

10:08

because the example comes from the US of hiring

10:10

people to go around the doctors

10:13

and annotating what they do. I

10:16

acknowledge that it leads to

10:18

another kind of criticism. Yeah,

10:22

that's like hiring a medical scribe

10:25

who essentially follows the clinician around,

10:28

better documenting, so that there's quote unquote

10:30

better data. There must be

10:32

another way besides adding one more

10:34

kind of human. It sounds unbelievably

10:37

impractical. Yeah. I

10:39

guess...

10:40

But it was interesting that... Sorry.

10:42

It was interesting that piece, the way they talked

10:46

about this concept of a learning health system,

10:49

which I think was interesting and they reference

10:51

that to being in the US. And the

10:54

idea I think being that

10:56

the clinicians that are working there are

10:58

seeing, I guess,

11:00

a much closer relationship between

11:03

research, quality improvement,

11:06

and their clinical care record and how

11:08

that information can be used simultaneously

11:11

for all of those things. And they did have some

11:13

examples in that, Joe, from the US. Do you know

11:16

much about those?

11:17

I think we've come a long way with

11:20

understanding

11:21

quality and performance in healthcare.

11:25

Whereas 20 years ago, we were really

11:27

at our infancy recognizing

11:29

that we need to find ways to measure

11:32

quality. There are certain evidence-based

11:34

processes that should be followed

11:36

for broad ranges

11:39

of conditions. But now, I

11:41

think we're at the point where we're

11:43

sort of at the avant-garde. WANDA is right. Better

11:45

think about how data can

11:47

be used to assess in a more

11:50

nuanced way what we're doing for

11:51

patients.

11:53

I think the key is less about

11:56

should we be monitoring to make sure there's checking

11:58

of a box that everybody gets

11:59

tobacco cessation or mental health

12:02

planning or whatever it is, knowing that those

12:04

types of interventions are easy to tick

12:06

but kind of probably

12:08

low impact because they're sort of

12:11

quickly sussed over in the course of a visit.

12:14

But instead, can we use

12:16

data to identify patients who I

12:18

like to think of as like falling through the cracks, people

12:20

who are completely missing from the system? Like

12:23

whether their blood pressure is 148 over 88 versus 152 over 92

12:25

is less important than

12:30

identifying the people who are completely untreated,

12:32

have known hypertension, they've been diagnosed with

12:35

it and they're not coming back for care

12:37

for whatever other reason. Maybe it's,

12:39

you know, the burden of care, maybe they're taking

12:41

care of another individual, maybe they can't

12:44

afford to take the day off from work, right?

12:46

These are the people that we need to reach as a health system

12:48

if we're going to improve population health.

12:51

Indeed, there was a nice editorial that,

12:53

going back to your research paper that we started this

12:55

discussion with, there was a nice editorial that goes

12:58

with it by Kath Cheklund, a

13:00

professor of health policy and primary

13:02

care in Manchester. And that gives a nice

13:04

overview, I think, of that whole quaff

13:07

endeavor where it

13:09

is sort of a bit historical where it came from. And

13:12

she summarises the extensive

13:15

evaluations that happened concluding

13:18

that the benefits were modest

13:20

at best. There was a lot of attainment

13:23

and achievement of these goals, but the

13:25

evidence suggested that the scheme, although

13:27

it had narrowed some inequalities in care

13:29

quality, the longer term evaluation wasn't

13:32

as good as was imagined.

13:34

And 10 years after its inception,

13:39

there was a decision that it hadn't really been associated

13:41

with the kind of improvements in mortality

13:43

and modelling suggested that

13:46

it wasn't cost effective. So things have

13:48

kind of moved on. It will be interesting

13:50

to see what comes next and maybe

13:52

something which feels much more suitable for 2023

13:57

and onwards, something that's much more patient centred.

14:00

and focused around the priorities that people have,

14:03

rather than the priorities that were

14:05

decided for them. By the powers

14:08

that be. The administrators. Exactly,

14:11

exactly.

14:17

You'll forgive

14:17

me for being a little bit UK-centric

14:20

on this item, but there was an announcement by

14:22

Chancellor Jeremy Hunt here on the 15th

14:25

of March about

14:27

drug and, well, medicines and

14:29

technology regulation in the UK. And

14:32

he said there was going to be near-automatic sign-up

14:34

for medicines and technologies already approved

14:37

by trusted regulators. And this is

14:39

part of setting out after

14:41

Brexit what's going to happen in

14:44

this space. Recently, I spoke

14:46

with Husse Naji

14:47

to tell us a bit more.

14:52

Hi, Hussein, thank you so much for joining me.

14:54

Can you just begin by introducing yourself

14:57

and giving a sense to our listeners

15:00

of why you are a man who knows stuff about this issue?

15:02

Thanks, Ellen. Thanks so much for having

15:04

me. I'm Hussein Naji. I'm an associate professor

15:06

of health policy at the London School of Economics and

15:09

Political Science, and my research

15:11

is on pharmaceutical policy and regulation. And

15:14

specifically, I look at the quantity and the quality

15:16

of evidence that supports regulatory

15:18

approvals of new medicines.

15:20

OK, so I think just

15:22

for any listeners who are less familiar with the

15:24

regulatory environment, just give us a swift

15:27

rundown on the key players. Can you just tell

15:29

us who the regulators are and

15:31

then who the health technology appraisers

15:33

are? Because I think that's going to put this conversation

15:36

in context for them.

15:38

Right. So traditionally, we would

15:40

have

15:41

private companies developing their products and then

15:43

seeking an approval by a regulatory

15:45

agency. And this can be an organisation

15:48

like the US Food and Drug Administration in the US

15:50

or the European Medicines Agency within

15:52

the European Union, or in the case of the UK,

15:55

the Medicines and Healthcare Products Regulatory

15:57

Agency, the MHRA.

15:59

products are approved by the regulatory agencies,

16:02

they're then appraised by, usually,

16:04

not in all countries, but usually they're appraised

16:06

by health technology assessment bodies. And in

16:08

the UK, this is nationally suited for

16:11

health and care excellence or NICE, that

16:13

looks at not only the efficacy of the product, but also

16:15

its cost effectiveness to see if it's good

16:18

value for money for the NHS.

16:20

So the regulators in effect answer,

16:23

might it work sufficiently well to

16:25

have a license to sell this thing

16:27

and the health care, sorry, and the health

16:29

technology appraisers are saying,

16:31

is it really worth it? That's exactly

16:33

right. Okay. So what does this announcement

16:36

change? So this announcement

16:38

essentially means that the MHRA

16:40

will be looking at other international

16:43

regulators and it's going

16:45

to be increasing its reliance on other regulatory

16:47

agencies for approving new medicines.

16:50

And why would it do that? So

16:54

much of the kind of the

16:56

announcement can be or the motivation for the

16:58

announcement can be traced back to Brexit,

17:01

which is of course, the UK's departure

17:03

from the European Union a few years ago.

17:06

Until Brexit, the UK was

17:08

part of the European regulatory framework.

17:11

And it was working under the auspices

17:13

of the European Medicines Agency. And

17:16

it was really a crucial kind of member

17:18

of the European Medicines Agency in that it was

17:21

responsible for over 30% of drug

17:23

reviews done by

17:24

the EMA. And it in turn,

17:26

it benefited from collaboration with

17:29

other regulatory agencies within the European Union.

17:32

And this also had a benefit for the industry

17:34

in that once a product was approved

17:36

by the European Medicines Agency, it could

17:38

be launched in all of the EU countries subsequently.

17:42

Since Brexit, the MHRA has become

17:44

the independent, the sole regulator for

17:47

the UK market. And what that means is

17:49

companies now have to seek

17:52

authorization from the MHRA before they can launch

17:54

their products in the UK. Now,

17:56

what this means is that there's a bit of

17:58

a concern from the government. a very

18:01

explicit concern from the government that UK

18:04

being a relatively small market, it only accounts

18:06

for 3% or 4% of

18:08

the global pharmaceutical market, companies

18:11

may not launch their products here or they may delay

18:13

launching their products here. So this announcement

18:16

is essentially sending a signal to

18:18

companies that the UK remains

18:20

an attractive, reliable market for companies

18:23

to launch their products in the UK.

18:24

And so what do you see as

18:27

the potential benefits and

18:29

harms of this announcement?

18:30

So it does have some

18:32

benefits in theory. Anytime

18:35

we have more kind of coordination or collaboration

18:38

or harmonisation of regulatory processes,

18:41

that's in theory a good thing. We

18:43

saw that with COVID-19 when there was

18:45

a bit of a lack of

18:47

a collaboration between regulatory agencies.

18:51

Companies couldn't agree on efficacy thresholds for vaccines,

18:53

for example. Some countries approved

18:57

the AstraZeneca vaccine, whereas others didn't.

19:00

So we could see benefits of better

19:02

collaboration. So

19:04

that's a positive thing about this

19:06

announcement that the MHRA will work closely with

19:08

other regulators. The other

19:11

potential benefits of this is that the MHRA

19:13

is going to get quite a bit of a funding boost

19:15

over the next few years. Okay. How's that work?

19:17

And it's going to receive

19:20

approximately £10 million for the next two years.

19:23

And that's really good because the MHRA

19:25

suffered really big budget cuts

19:27

following Brexit. And this led to staff

19:30

redundancies and some senior

19:33

staff members have really voiced concern

19:35

over what this may mean for the ability

19:38

of the organisation to really deliver on its

19:40

objectives. So that's a

19:42

positive. But on the flip side of this,

19:45

this additional funding is contingent

19:47

on speeding

19:47

up the approval

19:50

processes for the MHRA. So

19:53

MHRA is getting more money, but it's conditional

19:56

on speeding up its processes. And it's the

19:58

speed, this emphasis

19:59

on speed that is potentially

20:02

a problem as it can lead to all sorts

20:04

of unintended consequences for patients.

20:06

Tell us a bit about those. So

20:08

we know, for example, from the

20:11

US, which historically has had

20:13

a lot of experience

20:15

with speedy

20:17

drug review processes, when

20:20

regulators work under pressure and they work

20:22

under strict arbitrary rigid deadlines,

20:25

those approvals that happen immediately

20:28

before the deadline tend to have more

20:30

safety concerns that emerge during the

20:32

post-marketing period. So that's something that the

20:34

MHRA will need to really pay attention

20:36

to and monitor. In

20:39

terms of efficacy, there are some unintended consequences

20:41

as well, because when we

20:43

look at a drug early

20:45

on, then we may get a wrong

20:48

sense of how good this drug actually works if

20:50

we were to wait longer and get more mature data

20:52

about the drug. And one recent

20:55

example of this is the MHRA's

20:58

approval of molynepiravir

21:00

for high-risk COVID-19 patients.

21:04

The MHRA was the first regulator in the world to

21:06

approve this medicine, and it was based on

21:09

interim data from a clinical trial.

21:11

And that interim data made the drug look really

21:14

impressive in terms of reducing the risk

21:16

of hospitalizations and other severe outcomes.

21:19

But when more

21:21

mature data appeared from

21:24

the same clinical trial, the drug no longer

21:26

looked as effective as it initially appeared

21:28

on the basis of the interim data. So by looking at

21:30

drugs,

21:32

possibly prematurely, we may be

21:35

actually making suboptimal decisions

21:38

in terms of appraising the efficacy of products.

21:40

What if there's disagreement between the international

21:43

regulators? So FDA maybe said yes,

21:45

and EMA said no, or vice versa?

21:48

I think it's yet to figure out, but again,

21:52

it sounds like, again, this is trying

21:54

to read between the lines because there's no clear kind

21:56

of policy documents yet, that

21:58

we will

21:59

go for the speediest. So whichever one is

22:02

the first approval, we would adopt that

22:04

one because in Jeremy Hunt's words, I

22:06

mean, this is what he said and I quote,

22:08

this will put in place the quickest, simplest

22:11

regulatory approval in the world. So

22:15

if they approve it first, MHR is

22:17

going to adopt that decision is what it sounds like at

22:19

the moment, but it's unclear how it's going to be implemented

22:21

in practice.

22:22

And what are the external regulator gets it

22:24

wrong? Well, that's precisely

22:27

the concern. When we,

22:30

when the MHRA

22:32

looks to other regulators for their decisions

22:35

and almost automatically signs

22:37

off on them, then we may be subjecting

22:40

the NHS to really questionable

22:42

decisions from other regulators. And one

22:44

recent example of this is Adrucanumab,

22:47

which was approved by the US FDA

22:49

in 2021 for the treatment

22:51

of Alzheimer's disease. And this was

22:53

a very controversial decision because the

22:56

FDA gave the green light to this product

22:58

on the basis of a surrogate endpoint, which according

23:01

to established literature is

23:03

not really predictable, predicting delays

23:05

in cognitive impairment. And other regulatory

23:08

agencies actually refused approval

23:10

for this particular medicine, those in

23:12

Canada, Japan and the European Union.

23:15

So it's unclear at the moment what

23:17

safeguards the UK has in mind to

23:20

implement to prevent these types of

23:22

questionable approvals from getting an automatic

23:24

sign off in the UK.

23:29

So if you were in charge of the MHRA,

23:31

and you were thinking, how can I take

23:34

this forward in a way where we might

23:36

get,

23:37

maximize some of those benefits that

23:39

you mentioned, but minimize

23:42

the potential for harm? What do

23:44

they need to be thinking about or doing? I

23:46

think it would be really important to put as

23:49

much emphasis on the post-marketing

23:51

obligations or responsibilities of the regulator,

23:54

not only pre-market responsibilities.

23:56

There has been a lot of emphasis, as I mentioned, on

23:59

speeding up the approval.

23:59

of products, but this actually can

24:02

really backfire and it can have unintended

24:04

consequences because when you

24:07

get products onto the market quite early on

24:09

in their life cycle on the base of quite limited

24:11

or uncertain evidence, then you need to

24:13

really boost the resources available for

24:16

post-marketing responsibilities to make sure that

24:18

those drugs that appeared efficacious

24:20

initially, they continue to really

24:23

deliver benefits for patients in the

24:25

NHS or that they don't have post-marketing

24:27

safety concerns and if they do emerge then

24:29

these can be quickly spotted and then products

24:32

can be withdrawn if necessary and without

24:35

that type

24:37

of a more holistic approach

24:39

to drug regulation and putting all of the emphasis

24:41

on this early market access aspect

24:44

of it, I think this is

24:47

the aspect of the proposal or the

24:49

move that is potentially problematic.

24:51

So they need to keep an eye on what happens afterwards

24:54

and they need to be prepared then to change

24:56

their mind or adapt their position

24:58

as new evidence emerges.

25:03

So what do you do think of what Hussein had

25:05

to say? I thought

25:08

it was fascinating. It's obviously an interesting

25:10

and probably deliberately provocative

25:13

new policy from the MHRA,

25:15

this kind of concept of the SWIFT approval.

25:18

There's been a whole host of

25:20

research that's looked at

25:23

the issues around faster approval and

25:25

risk that Hussein alludes

25:28

to demonstrating that when products

25:31

are approved right near the deadlines there

25:33

does seem to be a greater risk that safety

25:36

issues identified later on. Do

25:38

you have a little interest to declare there, Joe?

25:41

Oh, well, that did come from a paper that

25:43

my group published and it built on a

25:45

sort of a seminal paper that Dan Carpenter published

25:48

two decades ago. And it's interesting,

25:50

we just published a paper

25:54

in BMJ evidence-based medicine that looked

25:56

at this issue where

25:58

kind of gets at why the

25:59

this may happen because what we identified is when

26:02

there were disagreements among the medical

26:04

officers, the reviewers at the FDA,

26:07

those applications take longer to decide upon,

26:10

but they are less likely to then have

26:12

a safety issue identified later on. And it could be

26:14

because they're sort of pressure checking and looking

26:16

at information. I mean, I'm speculating about why that

26:18

might be as opposed to the sort of rubber

26:20

stamp, everyone gets it out because it's got to make that

26:22

deadline. Obviously these

26:25

regulators will all benefit

26:27

from cooperation and there's a tremendous

26:29

amount of cooperation between the major

26:31

kind of high income regulators like the

26:34

US, Canada, EMA,

26:36

MHRA,

26:37

Japan and Australia already.

26:40

But this idea of kind of reciprocal approval,

26:42

kind of farming it out and allowing another

26:45

regulator to make the decision for your kind

26:47

of, you know, nation state is, it's

26:50

interesting. I mean, I think the biggest question

26:52

will be how does, what position does

26:54

this put NICE in? Because there's

26:56

a step of approval, regulatory approval,

26:58

demonstrating safety and effectiveness, and then there's a step

27:00

of, you know, are we going to pay for it at the cost, right?

27:03

In the US, right, we don't really have

27:05

that second step. So, you know,

27:07

generally approval means it's on

27:09

the market. And, you know, Hussein was

27:11

sort of getting at some of the challenges and

27:13

like what happened with Adjutana map.

27:16

But it's provocative. I don't know, Juan,

27:18

what do you think?

27:19

Yeah, that

27:21

was a shameless plug, wasn't it? But since

27:23

you mentioned BMJ, BM, we'll forgive

27:26

you. No,

27:28

it's a great paper, actually. It's

27:30

a great paper. But it was

27:32

fascinating to listen to Hussein analysis

27:36

of the situation. I guess for me, it

27:38

wasn't that shocking because many countries

27:41

in Latin America use

27:43

this process to sort

27:46

of triangulate the approval

27:49

in other stringent

27:51

regulatory agencies to fast

27:54

track products for many

27:56

reasons. The market being

28:00

not so attractive for companies to place their

28:02

products is one that Hussein already mentioned

28:04

and the situation in the UK seems

28:07

to be signaling that way. Another

28:10

perspective has to do with the reduction in

28:12

costs that relates to the

28:14

approval and most of these costs are usually

28:16

bared by the public

28:20

administration. So perhaps

28:23

I'm not so skeptical in that regard

28:26

and perhaps I would also like to add another

28:29

dimension that perhaps

28:29

time, when

28:32

we talk about bureaucracy and

28:35

processes, time, I'm

28:37

less concerned about time but considering

28:40

that one of the main concerns that we have been having

28:42

in the last few years is the changing criteria

28:45

for approval. So one of the

28:47

concerns was that the

28:49

FDA has been relaxing the criteria

28:51

to approve drugs. So whether

28:54

we could do that more efficiently or not

28:57

in time, I think

28:59

it's great

29:00

but I think that we need

29:02

to also keep a close

29:05

look as to what is the, where are

29:07

the criteria that all of the agencies

29:09

are using and whether there's

29:12

a systemic relaxation of criteria

29:14

across agencies that would also impact

29:16

the UK. That

29:16

is quite interesting Juan, that

29:19

over time it's like

29:21

there's a difference in terms of what

29:23

it means to be a drug which has

29:26

been allowed access to the market. How

29:29

much certainty or uncertainty

29:32

do we still have about the benefit

29:34

and harm of the drug and

29:36

maybe the ability of regulators

29:38

Hussein was talking about there to

29:40

go, to place greater

29:43

emphasis on the post-marketing, not

29:45

just studies but just sort of information and

29:48

willingness to fine-tune

29:50

the approval or even sometimes to

29:52

change what they said in light

29:54

of new information. You

29:56

don't have that sort of necessary

29:58

sense of circularity.

32:00

reference tools or textbooks

32:04

are. Tell us about it.

32:06

Well this article was published

32:08

at the UVM Journal

32:11

and it's very interesting because when

32:13

we talk about point of care apps,

32:16

of course when I was trained in medical school

32:18

we didn't have those, and

32:20

I'm not that old, but then

32:23

in residency we did, started

32:25

using all of this during

32:28

our patient encounters.

32:31

We couldn't remember

32:34

some criteria for some disease. You turn

32:36

to your phone and you look at one of these

32:38

apps. And

32:41

they've grown exponentially. So

32:44

what this group of authors did is provided

32:47

a thorough assessment of how these apps that

32:49

provide information

32:53

for clinical

32:55

content that is useful at point of care for

32:57

clinicians. What is the

32:59

rigor of their development? How evidence-based

33:02

they are? And

33:04

they looked at apps for Android

33:07

and iPhone. And

33:09

they included eight apps, UpToDate,

33:12

Dynamet, BMJ, Best Practice, They're

33:14

Pretty Guidelines, Medscape, Five Minute

33:16

Clinical Consult, Pathway Medical

33:19

Knowledge, and Ambos Medical Knowledge.

33:22

The assessment

33:25

is quite thorough and they looked at different aspects.

33:28

One had to do with editorial quality

33:30

and how evidence-based they are. And

33:34

one of the top rated apps

33:36

were UpToDate, Dynamet, and

33:39

BMJ, Best Practice in terms of editorial quality and

33:41

evidence-based methodology. Go BMJ!

33:47

This is an independent evaluation.

33:52

But

33:55

it's very interesting the differences. For

33:57

example, UpToDate had a greater

33:59

collection of apps.

33:59

coverage, so for example,

34:02

that is very interesting for the internists

34:04

who are always looking for the rare disease,

34:06

well, okay, well, on the

34:08

bedside of the patient.

34:12

But for example, DINAMED

34:15

included a more structured

34:18

way of grading the certainty

34:20

of evidence, which is sort of

34:22

relates to this concept of grade and so on.

34:25

So each of them have pros

34:28

and cons, and I think

34:30

that it would be useful to think about how

34:33

this market is growing and

34:35

what's the regulatory framework

34:37

for this in terms of this

34:40

apps are influencing

34:42

the decision that doctors are making, so we need

34:45

to make sure that they're evidence-based.

34:47

Jo, you were whooping in a

34:49

very US style there in the

34:51

middle. Did you want to pitch in first?

34:54

Well, I'm proud that

34:57

the BMJ best practice comes off

35:00

in this objective measurement as not

35:02

being terrible. It's actually considered pretty

35:04

good and reliable and of good quality.

35:07

I mean, I'll just say, a

35:09

couple of times a year, I'm on the wards with

35:12

house staff and that I'm also in clinics

35:15

working as a preceptor. It's amazing.

35:18

These point of care resources,

35:20

either access through a computer or through your

35:22

smartphone,

35:24

they've completely changed the way medicine

35:27

works. It used to be, I'm even older than Juan.

35:31

I think up to date was just being developed

35:34

when I was a resident and there

35:36

was one computer on the

35:39

floor of

35:40

the whole hospital ward that had

35:42

it because you had to have an individual subscription

35:44

and so people would kind of line up

35:47

to look up things, but really, we're all still using big textbooks.

35:50

We would lug around or that would be in

35:52

the team room that you would sort of

35:55

look through. Now, you

35:57

don't have to memorize as much because

35:59

you can... just kind of remind yourself

36:02

of all these things you've learned by looking it up, you

36:04

know, either on the internet or you know through the app

36:07

and you know, that's that's probably really

36:09

good so long as they're reliable accurate

36:11

and objective and evidence-based and so

36:13

it's actually quite important to have

36:16

these kinds of assessments

36:18

it is quite an interesting issue to consider

36:20

isn't it because Do you imagine

36:23

that if you rewound back to

36:25

the era of the textbook? We would have evaluated

36:28

how evidence-based textbooks were

36:30

Well, let's just say Evidence

36:33

is very different now. I mean right, you

36:36

know, there's so much more evidence.

36:38

There's better trials There's many

36:40

more trials some of which are not better. There's

36:43

just more observational studies. There's more

36:45

surveys There's more of everything which you

36:47

know enable it's created a far

36:50

bigger You

36:51

know basis of evidence to

36:53

guide decision-making, right? I mean Back

36:56

in the era of the you know, the

36:58

80s even into the 90s, right? You

37:01

know there weren't there there just wasn't

37:03

as much research. It moved more slowly. So as

37:05

things as things have sped up We

37:08

need resources like this to help synthesize

37:11

and aggregate in my opinion

37:13

Yes, no, I totally agree. I totally

37:15

agree. You have to have that but it's interesting

37:18

to think Going forwards if

37:20

if you're

37:21

if we collectively are our

37:24

EBM nerds We care about this thing and

37:26

we care that what people are doing is evidence-based

37:28

It's interesting to think how you can more

37:31

clearly communicate that information so that

37:33

you're sharing what amounts to sort

37:35

of received wisdom or things that we sort

37:37

of do as opposed to things

37:41

which through some clear evidence

37:43

framework we understand are

37:46

maybe worthwhile or Clearly

37:49

you need to do this thing based on best evidence

37:51

and I still don't think we've quite got that right I don't

37:53

think we've quite got the conversation about where

37:57

What you're reading what information that's based

37:59

on? as clear as

38:01

it could or should

38:02

be. I see exactly what you mean now, and

38:04

I totally agree, right? Because when UpToDate puts out

38:07

a sort of a recommendation or

38:09

a guidance, right, that doesn't necessarily

38:11

mean that it applies equally to any individual

38:13

patient, that the evidence that it was based

38:16

on, you know, is

38:18

relevant to older adults versus younger adults,

38:20

right? And so people with certain comorbid

38:22

diseases. And sometimes that does get lost,

38:25

right? Because you're not going to the source study, you're

38:27

just reading, you know, the synthesis of it.

38:30

So it's not to say that

38:32

that's not important. It just kind

38:34

of, I think it provides

38:37

a starting point, particularly

38:39

for those clinicians who want to follow

38:41

out the trail of evidence through links

38:44

embedded right within that point of source, point

38:46

of care source material. It's much easier

38:49

than of course, you know, sitting down and doing

38:51

a big PubMed search and then trying to manage that.

38:54

I think we all know that's

38:55

not going to happen. It's going

38:57

to be a clinical day. Right,

39:04

that's about all we have time for this

39:06

week. You can subscribe and rate

39:08

us wherever you get your podcast from

39:11

and we'll be back next

39:12

month with more from the world of evidence. Until

39:15

then, it's goodbye from me.

39:18

Goodbye from me. And goodbye from me. Take

39:21

care out there.