Journal of Soft Computing and Information Technology

Journal of Soft Computing and Information Technology

Improving Zero-Day Malware Detection Based on Ensemble Learning and Gaussian Kernel Fuzzy Rough Sets

Document Type : Persian Original Article

Authors
Akram Ansari 1
Javad Hamidzadeh 2
1 *- Faculty of Computer Engineering and Information Technology, Sadjad University, Mashhad, Iran.
2 Faculty of Computer Engineering and Information Technology, Sadjad University, Mashhad, Iran.
10.22034/jscit.2025.540111.2155
Abstract
Zero-day malware detection, due to the dynamic and unknown nature of these threats, is one of the major challenges in cybersecurity. The presence of mixed data comprising high-dimensional numerical and categorical features further complicates this problem. Such heterogeneity not only increases the computational burden but also blurs the decision boundaries between malicious and benign classes. Numerical features capture quantitative aspects such as file size or the number of sections, whereas categorical features represent qualitative characteristics such as file type or compiler signature. The simultaneous presence of these features makes feature selection difficult, as conventional methods lack sufficient capability to model nonlinear relationships combined with uncertainty between them, leading to degraded performance in zero-day malware detection .In this study, we propose a feature selection method based on Gaussian Kernel Fuzzy Rough Sets (GKFRS), where the selected features are utilized within a stacking ensemble learning framework to weight models according to their performance. Evaluation on the standard EMBER dataset demonstrates that the proposed method achieves a precision of 96% and a high recall rate, delivering effective detection of zero-day malware. The key contributions of this research include efficient feature selection, the integration of stacking ensemble learning, and comprehensive evaluation with credible results.
Keywords
Zero-day malware
Malware detection
Fuzzy rough sets with Gaussian kernel
Hybrid features
Ensemble learning
[1] M. S. Raza and U. Qamar, “An incremental dependency calculation technique for feature selection using rough sets,” Inf. Sci., vol. 343–344, pp. 41–65, May 2016, doi: 10.1016/j.ins.2016.01.044.
[2] J. Zhao, J. Liang, Z. Dong, D. Tang, and Z. Liu, “NEC: A nested equivalence class-based dependency calculation approach for fast feature selection using rough set theory,” Inf. Sci., vol. 536, pp. 431–453, Oct. 2020, doi: 10.1016/j.ins.2020.03.092.
[3] J. Zhang, Z. Luo, C. Li, C. Zhou, and S. Li, “Manifold regularized discriminative feature selection for multi-label learning,” Pattern Recognit., vol. 95, pp. 136–150, Nov. 2019, doi: 10.1016/j.patcog.2019.06.003.
[4] R. Huang and Z. Wu, “Multi-label feature selection via manifold regularization and dependence maximization,” Pattern Recognit., vol. 120, p. 108149, Dec. 2021, doi: 10.1016/j.patcog.2021.108149.
[5] L. Dong, R. Wang, and D. Chen, “Robust Feature Selection with Fuzzy Rough Set,” Int. J. Fuzzy Syst., July 2025, doi: 10.1007/s40815-025-02053-6.
[6] C. Wang, Y. Zhang, S. An, and T. Deng, “Adaptive feature selection based on fuzzy rough set fusion model with class variance,” Pattern Recognit., vol. 170, p. 112014, Feb. 2026, doi: 10.1016/j.patcog.2025.112014.
[7] M. Ebrahimifard, M. Abadi, and F. Deldar, “RansomHunter: zero-day Android ransomware detection using graph convolutional neural networks and code embeddings,” J. Supercomput., vol. 81, no. 15, p. 1351, Sept. 2025, doi: 10.1007/s11227-025-07777-8.
[8] M. T. Akdeniz, Z. Yeşilkaya, İ. E. Köse, İ. U. Ünal, and S. Şen, “VOLTRON: Detecting Unknown Malware Using Graph-Based Zero-Shot Learning,” 2025, arXiv. doi: 10.48550/ARXIV.2507.04275.
[9] A. Jain, R. Bagoria, and P. Arora, “An intelligent zero-day attack detection system using unsupervised machine learning for enhancing cyber security,” Knowl.-Based Syst., vol. 324, p. 113833, Aug. 2025, doi: 10.1016/j.knosys.2025.113833.
[10] A. Karakaya, “A Hybrid Approach for IoT Security: Combining Ensemble Learning with Fuzzy Logic,” Sensors, vol. 25, no. 18, p. 5668, Sept. 2025, doi: 10.3390/s25185668.
[11] Y. Xia, K. Chen, and Y. Yang, “Multi-label classification with weighted classifier selection and stacked ensemble,” Inf. Sci., vol. 557, pp. 421–442, May 2021, doi: 10.1016/j.ins.2020.06.017.
[12] I. Kononenko, “Estimating attributes: Analysis and extensions of RELIEF,” in Machine Learning: ECML-94, vol. 784, F. Bergadano and L. Raedt, Eds., in Lecture Notes in Computer Science, vol. 784. , Berlin, Heidelberg: Springer Berlin Heidelberg, 1994, pp. 171–182. doi: 10.1007/3-540-57868-4_57.
[13] M. Moradi and J. Hamidzadeh, “Ensemble-based Top-k Recommender System Considering Incomplete Data,” J. AI Data Min., vol. 7, no. 3, July 2019, doi: 10.22044/jadm.2019.7026.1830.
[14] N. X. Vinh, S. Zhou, J. Chan, and J. Bailey, “Can high-order dependencies improve mutual information based feature selection?,” Pattern Recognit., vol. 53, pp. 46–58, May 2016, doi: 10.1016/j.patcog.2015.11.007.
[15] Y. Li, T. Li, and H. Liu, “Recent advances in feature selection and its applications,” Knowl. Inf. Syst., vol. 53, no. 3, pp. 551–577, Dec. 2017, doi: 10.1007/s10115-017-1059-8.
[16] M.-L. Zhang and Z.-H. Zhou, “A Review on Multi-Label Learning Algorithms,” IEEE Trans. Knowl. Data Eng., vol. 26, no. 8, pp. 1819–1837, Aug. 2014, doi: 10.1109/TKDE.2013.39.
[17] H. Liu and H. Motoda, Feature Selection for Knowledge Discovery and Data Mining. Boston, MA: Springer US, 1998. doi: 10.1007/978-1-4615-5689-3.
[18] Y. Fan, J. Liu, W. Weng, B. Chen, Y. Chen, and S. Wu, “Multi-label feature selection with constraint regression and adaptive spectral graph,” Knowl.-Based Syst., vol. 212, p. 106621, Jan. 2021, doi: 10.1016/j.knosys.2020.106621.
[19] W. Qian, J. Huang, F. Xu, W. Shu, and W. Ding, “A survey on multi-label feature selection from perspectives of label fusion,” Inf. Fusion, vol. 100, p. 101948, Dec. 2023, doi: 10.1016/j.inffus.2023.101948.
[20] N. Spolaor, E. A. Cherman, M. C. Monard, and H. D. Lee, “ReliefF for Multi-label Feature Selection,” in 2013 Brazilian Conference on Intelligent Systems, Fortaleza, Brazil: IEEE, Oct. 2013, pp. 6–11. doi: 10.1109/BRACIS.2013.10.
[21] A. M. Alnajim, S. Habib, M. Islam, R. Albelaihi, and A. Alabdulatif, “Mitigating the Risks of Malware Attacks with Deep Learning Techniques,” Electronics, vol. 12, no. 14, p. 3166, July 2023, doi: 10.3390/electronics12143166.
[22] M. Ahmed, N. Afreen, M. Ahmed, M. Sameer, and J. Ahamed, “An inception V3 approach for malware classification using machine learning and transfer learning,” Int. J. Intell. Netw., vol. 4, pp. 11–18, 2023, doi: 10.1016/j.ijin.2022.11.005.
[23] A. Ksibi, M. Zakariah, L. Almuqren, and A. S. Alluhaidan, “Efficient android malware identification with limited training data utilizing multiple convolution neural network techniques,” Eng. Appl. Artif. Intell., vol. 127, p. 107390, Jan. 2024, doi: 10.1016/j.engappai.2023.107390.
[24] H. Deng, C. Guo, G. Shen, Y. Cui, and Y. Ping, “MCTVD: A malware classification method based on three-channel visualization and deep learning,” Comput. Secur., vol. 126, p. 103084, Mar. 2023, doi: 10.1016/j.cose.2022.103084.
[25] S. Ramesh, S. Selvarayan, V. M. Raghav, and C. Arumugam, “A Multistage Malware Detection and Classification Model Using Visual Features,” in 2023 4th International Conference on Communication, Computing and Industry 6.0 (C216), Bangalore, India: IEEE, Dec. 2023, pp. 1–6. doi: 10.1109/C2I659362.2023.10431264.
[26] K. Shaukat, S. Luo, and V. Varadharajan, “A novel deep learning-based approach for malware detection,” Eng. Appl. Artif. Intell., vol. 122, p. 106030, June 2023, doi: 10.1016/j.engappai.2023.106030.
[27] D. Vasan, M. Alazab, S. Wassan, H. Naeem, B. Safaei, and Q. Zheng, “IMCFN: Image-based malware classification using fine-tuned convolutional neural network architecture,” Comput. Netw., vol. 171, p. 107138, Apr. 2020, doi: 10.1016/j.comnet.2020.107138.
[28] J. Chen, C. Yuan, J. Li, D. Tian, R. Ma, and X. Jia, “ELAMD: An ensemble learning framework for adversarial malware defense,” J. Inf. Secur. Appl., vol. 75, p. 103508, June 2023, doi: 10.1016/j.jisa.2023.103508.
[29] Y. Gao, H. Hasegawa, Y. Yamaguchi, and H. Shimada, “Malware Detection by Control-Flow Graph Level Representation Learning With Graph Isomorphism Network,” IEEE Access, vol. 10, pp. 111830–111841, 2022, doi: 10.1109/ACCESS.2022.3215267.
[30] C. Joshi, R. K. Ranjan, and V. Bharti, “A Fuzzy Logic based feature engineering approach for Botnet detection using ANN,” J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 9, pp. 6872–6882, Oct. 2022, doi: 10.1016/j.jksuci.2021.06.018.
[31] X. Che, D. Chen, J. Deng, and J. Mi, “Exploiting local label correlation from sample perspective for multi-label classification via three-way decision theory,” Appl. Soft Comput., vol. 149, p. 110950, Dec. 2023, doi: 10.1016/j.asoc.2023.110950.
[32] A. S. Kumar and R. Rekha, “A Dense Network Approach with Gaussian Optimizer for Cardiovascular Disease Prediction,” New Gener. Comput., vol. 41, no. 4, pp. 859–878, Nov. 2023, doi: 10.1007/s00354-023-00234-1.
[33] L. K. Singh, M. Khanna, H. Monga, R. Singh, and G. Pandey, “Nature-Inspired Algorithms-Based Optimal Features Selection Strategy for COVID-19 Detection Using Medical Images,” New Gener. Comput., vol. 42, no. 4, pp. 761–824, Nov. 2024, doi: 10.1007/s00354-024-00255-4.
[34] M. A. Zarif and J. Hamidzadeh, “Improving performance of multi-label classification using ensemble of feature selection and outlier detection,” in 2022 12th International Conference on Computer and Knowledge Engineering (ICCKE), Mashhad, Iran, Islamic Republic of: IEEE, Nov. 2022, pp. 073–079. doi: 10.1109/ICCKE57176.2022.9960003.
[35] Y. Yang et al., “Rational linear kernelized weighted fuzzy rough attribute selection with class separability,” Fuzzy Sets Syst., vol. 521, p. 109600, Dec. 2025, doi: 10.1016/j.fss.2025.109600.
[36] C. Wang, Y. Zhang, S. An, and W. Ding, “An Adaptive Fuzzy Rough Neural Network and Its Application in Classification,” IEEE Trans. Syst. Man Cybern. Syst., vol. 55, no. 10, pp. 7579–7590, Oct. 2025, doi: 10.1109/TSMC.2025.3599882.
[37] J. Liu, M. Wu, M. Lin, and Z. Xu, “Anisotropic-Gaussian-Kernel-Based Fuzzy Clustering Algorithm for Feature Selection,” IEEE Trans. Fuzzy Syst., vol. 33, no. 9, pp. 3061–3075, Sept. 2025, doi: 10.1109/TFUZZ.2025.3581918.
[38] Q. Hu, L. Zhang, D. Chen, W. Pedrycz, and D. Yu, “Gaussian kernel based fuzzy rough sets: Model, uncertainty measures and applications,” Int. J. Approx. Reason., vol. 51, no. 4, pp. 453–471, Mar. 2010, doi: 10.1016/j.ijar.2010.01.004.
[39] M. Cen, X. Deng, F. Jiang, and R. Doss, “Zero-Ran Sniff: A zero-day ransomware early detection method based on zero-shot learning,” Comput. Secur., vol. 142, p. 103849, July 2024, doi: 10.1016/j.cose.2024.103849.
[40] Dept. of CSE, Rajarambapu Institute of Technology, Rajaramnagar, Sangli, Maharashtra, India, S. S. Lad., and A. C. Adamuthe, “Improved Deep Learning Model for Static PE Files Malware Detection and Classification,” Int. J. Comput. Netw. Inf. Secur., vol. 14, no. 2, pp. 14–26, Apr. 2022, doi: 10.5815/ijcnis.2022.02.02.
[41] N. M. Ali, M. A. Jabaar, and A. M. Taha, “A Dual-Branch CNN-LSTM Residual Network for Enhanced Windows Malware Detection,” J. Port Sci. Res., vol. 8, no. 4, July 2025, doi: 10.36371/port.2025.4.6.
[42] A. H. K, S. N S, and M. K. S D, “Enhancing Malware Detection: A Comparative Analysis of Ensemble Learning Approaches,” in 2024 First International Conference on Technological Innovations and Advance Computing (TIACOMP), Bali, Indonesia: IEEE, June 2024, pp. 35–40. doi: 10.1109/TIACOMP64125.2024.00016.
[43] M. Dener and S. Gulburun, “Clustering-Aided Supervised Malware Detection with Specialized Classifiers and Early Consensus,” Comput. Mater. Contin., vol. 75, no. 1, pp. 1235–1251, 2023, doi: 10.32604/cmc.2023.036357.
[44] H. V. Vo, P. H. Nguyen, H. T. Nguyen, D. B. Vu, and H. N. Nguyen, “Enhancing AI - Powered Malware Detection by Parallel Ensemble Learning,” in 2023 RIVF International Conference on Computing and Communication Technologies (RIVF), Hanoi, Vietnam: IEEE, Dec. 2023, pp. 503–508. doi: 10.1109/RIVF60135.2023.10471855.
Journal of Soft Computing and Information Technology
Volume 14, Issue 3
Autumn - October
Autumn 2025
Pages 61-76