Variable Selection under Multicollinearity and Heavy-Tailed Errors: A Systematic Assessment of Classical and Penalized Methods

Variable selection in high-dimensional regression becomes particularly challenging when multicollinearity and heavy-tailed errors occur simultaneously. This study systematically evaluates the performance boundaries of classical, shrinkage, sparse, and robust regression methods under these conditions. A Monte Carlo simulation experiment was conducted using 100 candidate predictors, of which 10 were truly active, while predictor correlation was varied across ρ = 0.1, 0.3, 0.6, and 0.9. Three error environments normal, contaminated normal, and Student-tt heavy-tailed errors were considered. Ordinary Least Squares (OLS), Ridge, LASSO, Elastic Net, and Huber regression were evaluated using mean squared error (MSE), mean absolute error (MAE), root mean squared error (RMSE), number of selected variables, true positive rate (TPR), false positive rate (FPR), coefficient variability, and selection instability. The findings reveal a clear trade-off between predictive accuracy, sparsity, robustness, and selection stability. OLS and Ridge achieved comparatively favorable predictive performance in several scenarios but retained all 100 predictors, resulting in substantial over-selection. LASSO achieved greater sparsity and relatively lower false-positive rates but became increasingly unstable under severe multicollinearity, with selection instability rising to 0.2976 at ρ = 0.9. Elastic Net demonstrated the most balanced performance, maintaining high signal recovery while substantially reducing selection instability as predictor correlation increased. Under contaminated errors, Elastic Net also outperformed LASSO in signal recovery and prediction. Huber regression reduced sensitivity to extreme observations but did not provide sparse variable selection. Overall, the results demonstrate that no conventional method simultaneously addresses heavy-tailed errors, multicollinearity, and sparse selection adequately. Elastic Net provides the strongest benchmark among the evaluated procedures, while the findings motivate the development of a robust, sparsity-inducing, and correlation-aware variable-selection framework for high-dimensional regression.

variable selection; multicollinearity; heavy-tailed errors; high-dimensional regression

1. Adomaityte, U., Defilippis, L., Loureiro, B., & Sicuro, G. (2024). High-dimensional robust regression under heavy-tailed data: Asymptotics and universality. Journal of Statistical Mechanics: Theory and Experiment, 2024(11), 114002. https://doi.org/10.1088/1742-5468/ad65e6 (DOI) [Google Scholar] [Crossref]

2. Amato, U., Antoniadis, A., De Feis, I., & Gijbels, I. (2022). Wavelet-based robust estimation and variable selection in nonparametric additive models. Statistics and Computing, 32, 11. https://doi.org/10.1007/s11222-021-10065-z (Springer Link) [Google Scholar] [Crossref]

3. Babarinsa, O., Edogbanya, H., Abari, O., & Adeniyi, I. (2025). Regression estimation and feature selection using modified correlation-adjusted Elastic Net penalties. FUDMA Journal of Sciences, 9(1), 29–40. https://doi.org/10.33003/fjs-2025-0901-2774 (FUDMA Journal of Sciences) [Google Scholar] [Crossref]

4. Bottmer, L., Croux, C., & Wilms, I. (2022). Sparse regression for large data sets with outliers. European Journal of Operational Research, 297(2), 782–794. https://doi.org/10.1016/j.ejor.2021.05.049 (Maastricht University) [Google Scholar] [Crossref]

5. Bunea, F. (2008). Consistent selection via the Lasso for high dimensional approximating regression models. In Pushing the limits of contemporary statistics: Contributions in honor of Jayanta K. Ghosh (pp. 122–137). Institute of Mathematical Statistics. https://doi.org/10.1214/074921708000000101 (ResearchGate) [Google Scholar] [Crossref]

6. Chamlal, H., Benzmane, A., & Ouaderhman, T. (2024). Elastic net-based high dimensional data selection for regression. Expert Systems with Applications, 244, 122958. https://doi.org/10.1016/j.eswa.2023.122958 (ScienceDirect) [Google Scholar] [Crossref]

7. Chatla, S. B., & Mandal, A. (2026). Robust variable selection in high-dimensional nonparametric additive model. Annals of the Institute of Statistical Mathematics, 78, 115–140. https://doi.org/10.1007/s10463-025-00939-8 (DOI) [Google Scholar] [Crossref]

8. Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), 1348–1360. https://doi.org/10.1198/016214501753382273 [Google Scholar] [Crossref]

9. Ghosh, A., Jaenada, M., & Pardo, L. (2024). Robust adaptive variable selection in ultra-high dimensional linear regression models. Journal of Statistical Computation and Simulation, 94(3), 571–603. https://doi.org/10.1080/00949655.2023.2262669 (Taylor & Francis Online) [Google Scholar] [Crossref]

10. Guo, X., Li, Y., & Hsing, T. (2025). Variable selection and minimax prediction in high-dimensional functional linear models. Statistica Sinica. https://www3.stat.sinica.edu.tw/statistica/fp/SS-2025-0151.html (Stat Sinica) [Google Scholar] [Crossref]

11. Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67. https://doi.org/10.1080/00401706.1970.10488634 (Taylor & Francis Online) [Google Scholar] [Crossref]

12. Huber, P. J. (1964). Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1), 73–101. https://doi.org/10.1214/aoms/1177703732 [Google Scholar] [Crossref]

13. Jiang, Y., Lu, H., Huang, X., & Jiang, R. (2025). Robust variable selection via the weighted elastic-net multi-step screening procedure. Hacettepe Journal of Mathematics and Statistics, 54(6), 2350–2362. https://doi.org/10.15672/hujms.1701976 (DergiPark) [Google Scholar] [Crossref]

14. Kepplinger, D. (2023). Robust variable selection and estimation via adaptive elastic net S-estimators for linear regression. Computational Statistics & Data Analysis, 183, 107730. https://doi.org/10.1016/j.csda.2023.107730 (ScienceDirect) [Google Scholar] [Crossref]

15. Kurata, S., & Hirose, K. (2026). Robust and consistent model evaluation criteria in high-dimensional regression. Journal of Statistical Planning and Inference, 242, 106358. https://doi.org/10.1016/j.jspi.2025.106358 (Kyushu University Pure Portal Site) [Google Scholar] [Crossref]

16. Lambert-Lacroix, S., & Zwald, L. (2011). Robust regression through the Huber's criterion and adaptive Lasso penalty. Electronic Journal of Statistics, 5, 101–120. [I recommend verifying the DOI before final submission.] [Google Scholar] [Crossref]

17. Li, G., Peng, H., & Zhang, J. (2012). Robust rank correlation based screening. The Annals of Statistics, 40(3), 1846–1877. https://doi.org/10.1214/12-AOS1023 [Google Scholar] [Crossref]

18. Li, H., Xu, X., Lu, Y., Yu, X., Zhao, T., & Zhang, R. (2022). Robust variable selection based on relaxed LAD Lasso. Symmetry, 14(10), 2161. https://doi.org/10.3390/sym14102161 [Google Scholar] [Crossref]

19. Liu, Z., Song, Y., & Cheng, Y. (2023). Robust variable selection with exponential squared loss for the spatial Durbin model. Entropy, 25(2), 249. https://doi.org/10.3390/e25020249 [Google Scholar] [Crossref]

20. Man, R., Tan, K. M., Wang, Z., & Zhou, W.-X. (2024). Retire: Robust expectile regression in high dimensions. Journal of Econometrics, 239(2), 105459. https://doi.org/10.1016/j.jeconom.2023.04.004 (ScienceDirect) [Google Scholar] [Crossref]

21. Meinshausen, N., & Bühlmann, P. (2006). High-dimensional graphs and variable selection with the Lasso. The Annals of Statistics, 34(3), 1436–1462. https://doi.org/10.1214/009053606000000281 [Google Scholar] [Crossref]

22. Muthukrishnan, R., & James, C. K. (2024). The effect of multicollinearity on feature selection. Indian Journal of Science and Technology, 17(35), 3664–3668. https://doi.org/10.17485/IJST/v17i35.1876 [Google Scholar] [Crossref]

23. Nan, R., Wang, J., Li, H., & Luo, Y. (2025). Robust variable selection via Bayesian LASSO-composite quantile regression with empirical likelihood: A hybrid sampling approach. Mathematics, 13(14), 2287. https://doi.org/10.3390/math13142287 (MDPI) [Google Scholar] [Crossref]

24. Park, E., Wong, R. K., Kwon, J., & Chu, V. W. (2024). A stable model for maximizing the number of significant features. International Journal of Data Science and Analytics, 17, 443–460. https://doi.org/10.1007/s41060-023-00494-6 [Google Scholar] [Crossref]

25. Song, Y., Han, H., Fu, L., & Wang, T. (2024). Penalized weighted smoothed quantile regression for high-dimensional longitudinal data. Statistics in Medicine, 43(10), 2007–2042. https://doi.org/10.1002/sim.10056 [Google Scholar] [Crossref]

26. Song, Y., Li, Z., & Fang, M. (2022). Robust variable selection based on penalized composite quantile regression for high-dimensional single-index models. Mathematics, 10(12), 2000. https://doi.org/10.3390/math10122000 [Google Scholar] [Crossref]

27. Su, P., Tarr, G., & Muller, S. (2024). Robust variable selection under cellwise contamination. Journal of Statistical Computation and Simulation, 94(6), 1371–1387. https://doi.org/10.1080/00949655.2023.2286316 (Taylor & Francis Online) [Google Scholar] [Crossref]

28. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x [Google Scholar] [Crossref]

29. Wang, H., Li, G., & Jiang, G. (2007). Robust regression shrinkage and consistent variable selection through the LAD-Lasso. Journal of Business & Economic Statistics, 25(3), 347–355. https://doi.org/10.1198/073500106000000251 (Taylor & Francis Online) [Google Scholar] [Crossref]

30. Wang, W., Liang, J., Liu, R., Song, Y., & Zhang, M. (2022). A robust variable selection method for sparse online regression via the Elastic Net penalty. Mathematics, 10(16), 2985. https://doi.org/10.3390/math10162985 [Google Scholar] [Crossref]

31. Wang, Y., & Karunamuni, R. J. (2022). High-dimensional robust regression with LqL_q-loss functions. Computational Statistics & Data Analysis, 176, 107567. https://doi.org/10.1016/j.csda.2022.107567 (ScienceDirect) [Google Scholar] [Crossref]

32. Wu, X., Liang, R., & Yang, H. (2022). Penalized and constrained LAD estimation in fixed and high dimension. Statistical Papers, 63(1), 53–95. https://doi.org/10.1007/s00362-021-01229-0 [Google Scholar] [Crossref]

33. Wu, X., Wang, M., Hu, W., Tian, G.-L., & Li, T. (2023). Penalized M-estimation based on standard error adjusted adaptive Elastic-Net. Journal of Systems Science and Complexity, 36, 1265–1284. https://doi.org/10.1007/S11424-023-1400-0 (DOI) [Google Scholar] [Crossref]

34. Ye, Y., Shao, Y., & Li, C. (2022). A sparse approach for high-dimensional data with heavy-tailed noise. Economic Research-Ekonomska Istraživanja, 35(1), 2764–2780. https://doi.org/10.1080/1331677X.2021.1978306 [Google Scholar] [Crossref]

35. Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2), 894–942. https://doi.org/10.1214/09-AOS729 [Google Scholar] [Crossref]

36. Zou, H. (2006). The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101(476), 1418–1429. https://doi.org/10.1198/016214506000000735 [Google Scholar] [Crossref]

37. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the Elastic Net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301–320. https://doi.org/10.1111/j.1467-9868.2005.00503.x [Google Scholar] [Crossref]

• The Impact of Ownership Structure on Dividend Payout Policy of Listed Plantation Companies in Sri Lanka
• Urban Sustainability in North-East India: A Study through the lens of NER-SDG index
• Performance Assessment of Predictive Forecasting Techniques for Enhancing Hospital Supply Chain Efficiency in Healthcare Logistics
• The Fractured Self in Julian Barnes' Postmodern Fiction: Identity Crisis and Deflation in Metroland and the Sense of an Ending
• Impact of Flood on the Employment, Labour Productivity and Migration of Agricultural Labour in North Bihar