Comparison of Non-Convex Variable Selection Criteria in High-dimensional Data with Count Response

to the large number of predictors and potential collinearity. While significant work has been done on selecting predictors from high-dimensional data with metrical covariates and Gaussian responses, there is a gap in comparing the performances of SCAD (Smoothly Clipped Absolute Deviation) and MCP (Minimax Concave Penalty) across different tuning parameters for count responses. Such comparisons, especially with k-fold cross-validation, are rare, highlighting the need for this study. Aim: This study aims to compare the effectiveness of SCAD and MCP in high-dimensional datasets where the response variable is a count, specifically for various tuning parameters and cross-validation techniques. Method: A simulation study was conducted using datasets with 250 predictors and a sample size of 100 modelled by a Poisson distribution (without overdispersion). The performance of SCAD and MCP was evaluated based on selection accuracy, cross-validation error, and computational efficiency. Results: The results showed that SCAD's performance at tested tuning parameters was similar to the standard tuning parameter (γ = 3.7) at 10-fold cross-validation only. For MCP, the variable selection performance at tuning parameters γ = 4 and γ = 5 was comparable to the standard (γ = 3), while other values differed significantly. Additionally, 10-fold cross-validation yielded the lowest cross-validation errors, and MCP selected the fewest predictors. Conclusion: Non-convex variable selection methods, particularly SCAD and MCP, are recommended for their robustness and accuracy in identifying relevant predictors and enhancing predictive capabilities in high-dimensional datasets with count responses.