Distributed Data Analytics Framework for Smart Transportation

As the amount data from IoT devices on transportation systems increases, developing a robust pipeline to stream, store and process data became critical. In this study, We explore prediction accuracy and computational performance of various supervised and unsupervised algorithms perform on distributed systems for developing a smart transportation data pipeline. Using a subset of New York City Taxi & Limousine Commission data, we evaluate Logistic Regression, Random Forrest Regressors and Classifiers, Principal Component Analysis, and Gradient Boosted Regression and Classification Tree machine learning techniques on a commodity computer as well as on a distributed system. Employing Amazon S3, EC2 and EMR, MongoDB, and Spark, we identify the conditions-data size and algorithm- under which the performance of distributed systems excel.