Last week, the limbic covered a study suggesting artificial intelligence programs could outperform oncologists at predicting survival of advanced cancer patients.

The Australian-led team had applied machine learning to analyse clinical text in oncology letters from initial consultations.

Using this, hey developed a model based on information from the oncologist’s first consultation letter that was a stronger predictor of 12-month survival than was ECOG performance status.

So is it time for human cancer specialists to hand over this aspect of their jobs to AI?

Oncologists and University of Sydney academics Professor Martin Stockler and Dr Belinda Kiely have spent years researching the topic and argue there is a long way to go.

the limbic: There can be a perception that oncologists struggle when it comes to predicting survival, but what does your research say?

Dr Belinda Kiely: Well we have data from multiple trials involving patients with incurable cancers, where oncologists have estimated the patients’ survival at enrolment and then the patients have been followed to see how long they actually live.

We’re planning to publish a review on these involving over 1000 patients and I think the main thing we have shown is that the doctors are just as likely to overestimate survival as underestimate it.

And their estimates are actually pretty well calibrated, particularly if you think about predicting a range for survival time rather than just one single point number.

Professor Martin Stockler: The other work that has been done is about how to express that estimate in a more meaningful way, rather than just saying ‘estimated survival time is x months’. Doing that, you can improve accuracy dramatically.

the limbic: So where does this idea that oncologists are terrible at estimating survival come from?

Dr Kiely: So there have been a lot of published studies looking at patients, often in palliative care settings, and in those cases there was a tendency for the doctors to overestimate survival. Of course, that was because patients were only a few days from death. But this led to a sort of dogma that doctors all overestimate survival.

But it is also the way we make our estimates. As Martin was saying, if you just come up with one number, it won’t be very precise. But if you give people a range of possible scenarios for their survival, then we know that that’s actually going to be very accurate.

the limbic: But that study we covered last week seemed to suggest AI could do it better?

Professor Stockler: Well firstly ECOG performance status is about 50 years old and it’s really not used as a prognostic tool. It’s known to have a strong association with prognosis but its initial purpose wasn’t to try to guess how long people were going to live for. It’s actually about classifying whether people will be suitable for cytotoxic treatments.

And the AI wasn’t entirely artificial in the study, it was looking at what the doctors put in their letters and what they thought was important enough to convey to another doctor. It would have summarised what they thought the patient’s condition and status was, what treatments they were going to do and what tests they had done. It’s not as though it was secret information. And it would be a miracle if that information wasn’t associated with the person’s prognosis.

It’s kind of interesting that even the crude ECOG performance status scale provided additional information above and beyond what was in the letter, which tells you the information being pulled out wasn’t that sophisticated.

What I would love to see would be a study that actually looked at how AI compared with what the doctor thought and my guess is both would give important prognostic information.

the limbic: How common is it for oncologists to actually discuss the prognosis with patients and when do those conversations usually take place?

Professor Stockler: When you ask oncologists, they’ll usually say something like 50-70% of patients will ask that question at some point. Of course, it’s not as though they are asking at every appointment, but some people will ask at the very beginning, other people ask when they are changing treatment or something else is changing.

the limbic: Given the importance of this question to patients, should there be formal processes for giving survival estimates embedded into care pathways?

Dr Kiely: We surveyed about 500 people with cancer about would they would like to be informed about their survival time. About 90% said they would prefer a format with their median survival as well as their best case and worst case scenarios.

The other thing to keep in mind is a lot of people will just go onto Google for an estimate on their survival after a cancer diagnosis, meaning they are getting information that is potentially not accurate.

Maybe they’re too scared to ask their doctor so the conversation doesn’t happen initially. So another thing we have been doing is working on a web-based tool to get them better answers.

Professor Stockler: It involves about 5 million US patients with a range of different cancers and gives you all sorts of information.

The first thing is to ask whether the patient wants any information about their prognosis and, if they do, how that would like that information conveyed. Not everybody wants numbers. Some people want just timeframes, or even just some idea of whether it’s curable.

Typically a first consultation will involve 20 or 30 different uncertainties and bits of information, so my preference is to allow the patient to absorb whatever they want to know first. It might be the second or third consultation where they have enough information where they’ve got enough information to be prepared for that question.

So I think it has to be a tailored approach according to what the person is ready for and what they want. Our work involved trying to equip doctors with a framework for being able to estimate the prognosis accurately and to explain it in a way that makes sense.

At the same time it’s about trying to encourage patients and their families to feel comfortable enough to ask that question and expect to get an answer that isn’t either simplistic or misleading.

the limbic: Do you think AI could be of some use?

Dr Kiely: Well I’m not sure about AI but certainly there is lots of use for the online tool we were talking about earlier based on those millions of US patients. What we’re saying to doctors is that if they don’t know what numbers to use, they could put in the cancer type and the patient’s age and some other basic demographic information and get a pretty good idea of their likely survival time.

At the moment I think that’s a better tool to come up with numbers than what the machine learning is up to. But maybe some time in the future, who knows?

Professor Stockler: The AI could be interesting, particularly if it pulled out things like their billing or postcode and all sorts of other random bits of information that you wouldn’t normally think was relevant, although in this study, the input was entirely the doctor’s letter.

the limbic: What’s the future of this? Where is the research likely to go to improve survival prediction and make it more of an exact science?

Dr Kiely: I don’t know. With a survival estimate, you’re never going to be able to predict it perfectly because it is all very uncertain. I think we can improve some things, but we’re never going to get to the point where I’ll say six months and a person lives exactly that long. There is always going to be a range around it.

We probably need to be working more on trying to help doctors have these conversations with patients and working out ways to help those conversations happen.

In terms of making things more accurate, one option could be to feed back information to doctors about the accuracy of their estimates and give them an idea of two they are doing, particularly around whether they are likely to over- or under-estimate.

Professor Stockler: That would be a pretty big loop if it was to happen, given with the studies we looked at, the typical survival time is a year – so it would take a year or longer to get information back about how good a particular estimate was. By then, you’ve probably forgotten the details of the patient at that point in time.

But I think what Belinda’s work shows is that if you express estimates in the way that we have recommended, they are remarkably accurate.

So we say that what we call the typical scenario is the middle 50% of patients. In our 1200-patient paper that’s about to be reported, 56% of patients were in that range.

Worst case scenario is when we say 5-10% of patients would have died, and that was about 11%. We were almost exactly right with our best case scenario estimates as well.

The point is that survival is inherently unpredictable and has a substantial random element. There’s actually some really nice statistical research showing that no matter how good your models are, that random element can’t be removed. So it’s a bit unrealistic to think you could even come up with a point estimate that’s going to be accurate.