Using Spark for Text Mining on Large Scale Liver Cancer Literature

Cancer is one of the main causes of death. The number of known cancer to-date is more than one hundred. Liver cancer, ranked after trachea and lung cancer, has long been high in the cancer leading cause in Taiwan and even ranked as second in 2016. The number of scientific articles related to cancer proliferates every year. The number reaches as high as 20 million in PubMed so that discovering useful information from the massive collection is very difficult. In addition, using a single machine to sift through these articles is very time-consuming. Therefore, we present a big data analytic framework using the distributed Apache Spark platform for text mining in PubMed literature. We establish a prediction model for liver cancer articles so that researchers may effectively validate whether an article is related to liver cancer or not. Classification models in Spark MLlib including Linear Support Vector Machines (SVM), Logistic Regression, are used in our experiments. Relevancy to liver cancer is further confirmed by using MeSH (Medical Subject Headings) terms. Logistic regression is about 3 times faster than SVMs and the accuracy of both methods is close to 95% in the experiments using hold-out validation. When maxfeatures is 500 and mindf ≤ 0. 1 (or mindf = 1), the accuracy may reach 96%. In the experiments with K-fold cross-validation, the accuracy of SVMs methods is 96%. The experimental results show that that our prediction model may effectively classify liver cancer articles.