Piloting IBM Watson Oncology within Memorial Sloan Kettering’s regional network.

e17653 Background: We have reported on efforts to develop IBM Watson Oncology, a cognitive computing application designed to inform clinical decision making. The initial prototype demonstrated results in lung and breast cancers, and addressed some complexities of natural language processing (Bach, Proc ASCO 2013). Here, we report our experience with a pilot testing program with staff at Memorial Sloan Kettering’s (MSK) regional network who are not part of the development and training team. Methods: Medical oncologists from MSK’s regional network were invited to evaluate the application. These physicians manage patients with many primary types of cancer. We gave login credentials for the internet-based application and asked each doctor to use the program designed to assist in choosing initial therapies for breast and colorectal cancers. Members of the development team, including physicians, analysts, engineers, and usability experts interviewed these pilot users to gather feedback. Data from each user’s selection of treatment options was collected, including assessments of accuracy and quality. Users could report issues directly through an embedded tool. Results: To date, six users have independently logged into the pilot, created and submitted cases, and evaluated the application’s output and functionality. Users recognized value in selecting appropriate treatment options, but noted opportunities to enhance the usability and utility of the pilot for physicians using the tool as part of their normal routines. Users commented that it took too long to enter a case because some of the approximately 20 clinical attributes were irrelevant and felt data should have been imported from existing data sources. Users also requested the device provide more information on how treatments were scored and ranked. Conclusions: IBM Watson Oncology’s development team has created an initial pilot application. We plan to improve the application’s usability and utility by enhancing integration with electronic medical record systems, expanding users’ ability to iteratively add new clinical attributes, and providing additional information on the factors influencing the system’s output. Testing by MSK medical oncologists and other organizations continues.