Improving Performance and Accuracy in Decision Trees: A Literature Survey on Impurity Functions

Decision tree algorithms have remained some of the most popular tools for supervised learning tasks. This has been because of their comprehensibility, malleability, and robustness to accommodate varying types of data. Nevertheless, their accuracy and performance are dependent on the use of impurity measures to guide the process of tree splits. The standard impurity measures, such as the Gini impurity, Entropy, and Classification Error, tend to face problems as the complexity, imbalance, and noise levels are raised. This often gives rise to overfitting. An examination of the disadvantages and present research efforts on impurity-based optimization of decision trees, as well as the study of new paradigms such as the use of Rényi Entropy, the utilization of the Tsallis Entropy, combinations of impurity functions, complexity-aware tree splits, and tailored impurity function augmentation, makes it clear that there are efforts underway to improve the accuracy, robustness, and comprehensibility, as well as the processing complexity, for the tasks of decision tree algorithms.