In June 2018, Babylon Health hosted an event in London at which it showed off its latest digital healthcare development. It had developed artificial intelligence (AI) that it claimed was “better than a doctor”.
Considered a world-first, the AI proved it was on par with practising clinicians by taking tests, including a set of questions from the MRCGP exam – a test that has to be taken by every GP in the UK. Scoring higher than the average score over a period of five years, the AI achieved 81% during its first sitting.
The industry and media alike were abuzz with excitement. Imagine – in the years to come, patients would be speaking with AI doctors instead of human ones. However, what followed would raise some serious questions about AI in healthcare.
Babylon – a Clean Bill of Health?
Professor Enrico Coiera, director of the Centre for Health Informatics at the Australian Institute of Health Innovation at Sydney’s Macquarie University, published a short peer review on Twitter in response to the conference paper from Babylon. Highlighting some issues with the test, he commented that, rather than being an AI, the “Babylon engine is a Bayesian reasoner” and the scenario wasn’t naturalistic.
“In summary, this is a very preliminary and artificial test of a Bayesian reasoner on cases for which it has already been trained,” said Professor Coiera. “In machine learning, this would be roughly equivalent to in-sample reporting of performance on the data used to develop the algorithm. A good practice is to report out-of-sample performance on previously unseen cases.”
It is through this thread that other advocates for good governance came forward to raise their own concerns about the company. Under the pseudonym Dr Murphy, a consultant from the UK has been vocal about their concerns over chatbots and AI being used by the NHS, and had reported Babylon Health to the Medicines and Healthcare Products Regulatory Agency (MHRA). Dr Selena Singh, of Guy’s & St Thomas’ NHS Foundation Trust, also shared results from the app on Twitter that showed a woman of childbearing age with a late period and left iliac fossa pain who had been diagnosed with an inguinal hernia, as opposed to the obvious differential diagnoses of ectopic pregnancy or appendicitis.
“Under the pseudonym Dr Murphy, a consultant from the UK has been vocal about their concerns over chatbots and AI being used by the NHS, and had reported Babylon Health to the Medicines and Healthcare Products Regulatory Agency”
Babylon Health has responded and defended itself since that time, but the doubt remains. Then, in December 2018, Forbes published an exclusive about doctors flagging to the company’s CEO problems with Babylon’s chatbot – the same one it had shown off in London six months later.
So, should we be concerned about AI coming into healthcare? Or, on the contrary, is it in fact the future?
Saving Lives – or Risking Them?
“We should discount the word ‘artificial’ when we talk about how artificial intelligence can save the lives of sick people – let’s use the more useful term machine learning,” says Dan Worman, CEO of Refero.
“Your local hospital probably has about 150 distinct software applications running right now and each one contains important data that could save or change a life, so let’s call it the hospital’s total ‘intelligence’. There’s no doctor, nurse or midwife in that hospital who can access all 150 all at once, and use what the hospital really ‘knows’ about an illness, a crisis, or a patient. Machine learning could connect the knowledge and present the statistical information that no clinician will ever be able to compute themselves. The hospital’s intelligence is no longer simply artificial – it becomes real.”
“Machine learning could connect the knowledge and present the statistical information that no clinician will ever be able to compute themselves. The hospital’s intelligence is no longer simply artificial – it becomes real”
This intelligence can be used to “build bridges between healthcare and other life-changing public services” such as social care, policing or mental health services, according to Worman. In an environment that still relies on the dated fax machine, this is a welcome innovation.
Just How Safe is Safe?
DoctorLink, a healthcare startup focusing on machine learning and data analytics, is one company that is taking the safety of its product very seriously. However, it admits that the regulation is behind the technology.
“Standards are a bit of a woolly area,” Ben Littlewood-Hillsdon, Chief Clinical Content Officer of DoctorLink, tells me. “It’s growing with the regulator – the Care Quality Commission.” He goes on to explain that DoctorLink has decided to ensure its own algorithms and processes are robust and ethically safe.
“We have a panel of doctors, nurses, scientists and others who look at our processes,” he explains. “Traceability is needed in healthcare – it’s not just about efficiency.” The company is, as far as it knows, the only healthcare start-up to have its technology underwritten by an insurance company, meaning GPs and other clinicians will not take the blame if something goes awry with DoctorLink’s platform.
“We have a panel of doctors, nurses, scientists and others who look at our processes. Traceability is needed in healthcare – it’s not just about efficiency”
According to Babylon Health’s product-safety information sheet, “all conversations with the service are logged, allowing full traceability of any feedback or complaints”, and there have been no reported incidents of patient harm to date since 2016. It also outlines how its technology and products are compliant with NHS England and the MHRA.
One question from this is how we can ensure all technology, including chatbots, can be traced? Is there a more sophisticated way of doing it, rather than just keeping logs of conversations?
Could Blockchain Offer the Cure?
One idea is blockchain. In February 2018, Lydia Torne of Simmons & Simmons LLP, wrote for MedTech Insight on how blockchain could help with med-tech traceability. In her analysis, she outlines how the technology has revolutionised the financial sector and how she believes it could do the same for life sciences and the medical-technology supply chain.
“Blockchain could allow real-time recording of each stage of the supply, manufacture and distribution process, up to the point the product reaches the end-purchaser,” writes Torne. But she does go on to highlight the issues with blockchain, including that the data on the ledger is not private. In a setting such as the NHS, in which patient confidentiality is key and General Data Protection Regulation legislates data privacy, it doesn’t seem a feasible option.
What is clear is that AI, while welcomed by some healthcare professionals, isn’t ready to take over from doctors and nurses just yet. Until it is tested in the same circumstances as medical-school trainees, it should not be allowed to share the same accountability that thousands of UK clinicians take on today.
AI is being treated, quite rightly, with serious caution and will need to be proven before it sees widespread adoption and faith. The same can be said for most technologies that enter the healthcare industry. When public health or the healthcare system itself is at stake, we need to be sure that we can rely entirely on any tech we use.
One example is the phenomenon of perfectly healthy people booking an appointment with their doctors because their Apple Watch or Fitbit device wrongly told them that they were unhealthy. A report from the Academy of Medical Royal Colleges found that the phenomenon was such that the waves of “worried well” could overwhelm the health service and cause “harm at scale.” Further clogging of an already stretched health service may seem relatively trivial, too, when you consider the potential for misfiring tech to impact someone’s health. AI is being introduced cautiously, and this is a very good thing.
Illustrations by Kseniya Forbender
To contact the editor responsible for this story:
Margarita Khartanovich at [email protected]
- Artificial Intelligence Isn’t Ready to Take Over From Doctors and Nurses, Just Yet
- Machine Learning Vs. Analysts: Will AI Eventually Replace Data Scientists?
- Why The Danger of Deepfakes Is No Danger At All
- Sarcastic Robots? How Deep Convolutional Neural Networks Are Making AI Worryingly Human
- Traffic Lights in The Sky: Flying Cars Can Appear Sooner Than You Think