M. D. Anderson Breaks With IBM Watson, Raising Questions About Artificial Intelligence in Oncology

In 2012, the University of Texas M. D. Anderson Cancer Center in Houston partnered with IBM to develop the artificial intelligence program, called IBM Watson, as a clinical decision tool in oncology. Five years and 62 million later, M. D. Anderson let its contract with IBM expire before anyone used Watson on actual patients. Last February, a university audit of the project exposed many procurement problems, cost overruns, and delays. Although the audit took no position on Watson’s scientific basis or functional capabilities, it did describe challenges with assimilating Watson into the hospital setting. Experts familiar with Watson’s applications in oncology describe problems with the system’s ability to digest written case reports, doctors’ notes, and other text-heavy information generated in medical care. That type of unstructured data differs from the structured data entered into drop-down boxes and other point-and click fields in an electronic medical record that Watson can more readily interpret. “The M. D. Anderson experience is telling us that solving data quality problems in unstructured data is a much bigger challenge for artificial intelligence than was first anticipated, ” said Amy Abernethy, M. D. , chief medical officer at Flatiron Health, a New York–based health care technology company, and former director of cancer research at the Duke Cancer Institute in Durham, N. C. Watson is famous for beating human contestants on Jeopardy in 2011. It can read 800 million pages per second, and it draws on the combined contents of PubMed, the National Cancer Institute’s Drug Dictionary, the Sanger Institute’s Catalogue of Somatic Mutations in Cancer database, every study registered on clinicaltrials. gov, and many other resources. IBM Watson Health, a specialized division commercializing the system in genomics and drug discovery, as well as oncology, has relationships with roughly 50 other institutions. “The M. D. Anderson experience is telling us that solving data quality problems in unstructured data is a much bigger challenge for artificial intelligence than was first anticipated. ” Watson’s selling point is that it helps doctors stay current with the volume of new findings published every day. IBM has been working on natural language processing capabilities that allow Watson to increasingly understand human speech. Oncologists have been trying to teach the computer system to think like a cancer doctor by training it with real and made-up cases. At M. D. Anderson, those training exercises started with a pilot focused on leukemia. But medical diagnosis is more complicated than Jeopardy questions. Institutions use medical terms in different ways, and despite the best efforts of software engineers, Watson still can’t interpret medical language as well as humans can. At M. D. Anderson, for instance, Watson couldn’t reliably distinguish the acronym for acute lymphoblastic leukemia, ALL, from the shorthand for allergy, which is often also written ALL. “To do that correctly, Watson would have to interpret those written characters in the appropriate context, ” Abernethy said. “It would have to know that ALL in the context of high white blood cell counts or a bone marrow transplant means leukemia and not something else. ” M. D. Anderson suspended its leukemia pilot midstream before switching to lung cancer “because project leaders thought that area would provide greater opportunity for a timely completion, ” the UT audit stated. Andrew Seidman, M. D. , a medical oncologist at the Memorial Sloan Kettering Cancer Center in New York, has been trying to teach Watson how to treat breast cancer. He said that to bring its accumulated knowledge to bear on treating patients at the hospital, Watson needs to be better integrated with electronic medical records. “The more efficient way to do that would be for it to extract medical attributes automatically, ” Seidman said. “But many of the attributes we want Watson to digest aren’t found in categorical structured data—they’re buried in narrative form in doctors’ consultation notes. ” And though natural language processing is central to Watson’s functionality, Seidman said that “it remains a work in progress. ” Andrew Norden, M. D. , deputy chief health officer at IBM Watson Health, acknowledged the need to clarify abbreviations in unstructured medical notes and “other things Watson hasn’t seen before. ” He added, “But this isn’t rocket science—the trick is getting the right digital content to Watson so that Watson can read it. ” In M. D. Anderson’s case, a third party—the London-based consulting firm PricewaterhouseCoopers—was contracted to develop a bioinformatics tool to integrate Watson with the hospital’s record system. According to M. D. Anderson’s audit, Watson’s treatment recommendations during the lung cancer pilot agreed with those of its human teachers nearly 90% of the time. “This is a very high level of accuracy, ” Norden said. Abernethy, however, was more skeptical. “What does 90% accuracy really mean? ” she asked. “Does that mean that for common, run-of-the-mill clinical scenarios the technology was wrong 10% of the time? Or does it mean that 10% of the time Watson couldn’t help on the more difficult cases for which treatment decisions may not be so clear-cut? ” Seidman said that in training the system, he’s trying to make Watson’s recommendations agree more often with practice standards at Sloan Kettering. But he said the hope is that Watson will eventually pick better treatments that he and his colleagues might not have considered without the technology. “What we’re really looking for are computer-assisted decisions that extend the time to disease progression and improve overall survival rates, ” he said. Harpreet Singh Buttar, a health care technology industry analyst with Frost otherwise, we’re structuring data at the expense of good medical care. ” Still, she added, “I do think that artificial intelligence systems will improve, as long as the unstructured data formats improve too. ”