Journal of Soft Computing and Information Technology

Journal of Soft Computing and Information Technology

Application of Approximate Computing Methods in Convolutional Neural Networks to Improve Accuracy and Performance

Document Type : Persian Original Article

Authors
Department of Electrical and Computer Engineering, Kharazmi University, Tehran, Iran.
Abstract
Artificial neural networks are one of the most influential and important branches of science in the past decades, which have been used in a wide range of applications such as medical and pharmaceutical services, voice and speech recognition, computer vision, natural language processing, video and image processing, etc. . In order for neural networks to have the most efficiency, the algorithms used in them must have deep networks with many layers, which requires very high computing power both in the network training stage and in the inference stage. In recent years, approximate computing has been one of the most promising approaches to reduce energy consumption in applications that can tolerate accuracy reduction. In the proposed method, specific to the first layer of the convolution neural network, in order to compress and reduce the computations, each pixel is given a label using the K-Means clustering algorithm on the input images, and based on that, the convolution process is performed using the average of the clustered pixels, which it will reduce the number of multiplication operations by at least 90%. Then, in order to further compression, using one of the neural network pruning methods based on the L1-norm criterion, different amounts of pruning are applied to all the layers of that network. According to the results of this research, the criteria of accuracy, efficiency, and parameter size of convolutional neural networks have been improved using the proposed method on 2 separate convolutional neural networks and on 4 different datasets.
Keywords

[1] N. Manikandan, M. Priyanka2, Sasikumar, R. Muthaiah, "Approximation Computing Techniques to Accelerate CNN Based Image Processing Applications – A Survey in Hardware/Software Perspective", International Journal of Advanced Trends in Computer Science and Engineering, Volume 9, No.3, 2020
[2] Q. Xu, T. Mytkowicz and N. Sung Kim, "Approximate Computing: A Survey", IEEE DESIGN & TEST, 2015
[3] G. Rodrigues, F. L. Kastensmidt, and A. Bosio, "Survey on Approximate Computing and Its Intrinsic Fault Tolerance", Electronics, 2020
[4] J. Koomey, S. Berard, M. Sanchez, H. Wong, "Implications of historical trends in the electrical efficiency of computing", IEEE Ann. Hist. Comput. 33(3), 46–54, 2011
[5] A. Be´rut, A. Arakelyan, A. Petrosyan, S. Ciliberto, R. Dillenschneider, E. Lutz, "Experimental verification of Landauer/’s principle linking information and thermodynamics", Nature 483(7388), 187–189, 2012
[6] H. Esmaeilzadeh et al., "Neural acceleration for general-purpose approximate programs", MICRO, 2012
[7] H. B. Barua, K. C. Mondal, "Approximate Computing: A Survey of Recent Trends—Bringing Greenness to Computing and Communication", The Institution of Engineers (India), 2019
[8] A. Sampson, W. Dietl, E. Fortuna, D. Gnanapragasam, L. Ceze, and D. Grossman. EnerJ: Approximate Data Types for Safe and General Low-Power Computation", In Proc. International Conference on Programming Language Design and Implementation (PLDI), pp. 164-174, 2011.
[9] J. Bornholt, T. Mytkowicz, and K. S. McKinley, "A first-order type for uncertain data", In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, USA, 2014
[10] G. G. Henry, T. Park, R. E. Hooker, "VIA Alliance Semiconductor Co Ltd", 2019
[11] V. Pejovic, "Towards Approximate Mobile Computing, Newsletter, GetMobile: Mobile Computing and Communications, vol. 22, no. 4, 2018
[12] Z. Peng, X. Chen, C. Xu, N. Jing, X. Liang, C. Lu, L. Jiang, "AXNet: ApproXimate computing using an end-to-end trainable neural network", in Proceedings of the International Conference on Computer-Aided Design, p. 11, ACM, 2018
[13] A. Mant, R.S. Yazdanbakhsh, A. Park, J. Thwaites, B. Esmaeilzadeh, H. Hassibi, A. Ceze, L. Burger, "General-purpose code acceleration with limited-precision analog computation", In Proceedings of the International Symposium on Computer Architecture, 2014
[14] E. Wang, J. J. Davis, P. Y. K. Cheung, and G. A. Constantinides, “LUTNet: Rethinking Inference in FPGA Soft Logic”, 2019 IEEE 27th Annu. Int. Symp. Field-Programmable Cust. Comput. Mach., pp. 26–34, 2019
[15] S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning bothWeights and Connections for Efficient Neural Networks,” Advances in neural information processing systems, pp. 1135–1145, 2015
[16] H. Jiang, C. Liu, L. Liu, F. Lombardi, and J. I. E. Han, “A Review, Classification, and Comparative Evaluation of Approximate arithmetic circuits”, ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 13, no. 4, pp. 1–34, 2017
[17] M. Imani, M. Samragh, Y. Kim, and S. Gupta, “RAPIDNN: In-Memory Deep Neural Network Acceleration Framework”, arxiv preprint arxiv, 2018
[18] D. Hammerstrom, “A VLSI architecture for high-performance, low-cost, on-chip learning”, 1990 IJCNN International Joint Conference on Neural Networks, 1990
[19] H. Li, H. Samet, A. Kadav, H. P. Graf, I. Durdanovic, "Pruning Filters for Efficient Convnets", International Conference on Learning Representations (ICLR), 2017
[20] A. Krizhevsky, G. Hinton, "Learning Multiple Layers of Features From Tiny Images", Technical report, University of Toronto, https://www.cs.utoronto.ca/~kriz/ learning-features-2009-TR.pdf.2009, 2009
[21] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, "Reading Digits in Natural Images with Unsupervised Feature Learning", NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011
[22] S. Li, W. Leng, "Real World Expression Recognition: A Highly Imbalanced Detection Problem", 9th IAPR International Conference on Biometrics (ICB), 2016
[23] K. Simonyan, A. Zesserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", International Conference on Learning Representations (ICLR), 2015
[24] K. He, X. Zhang, S. Ren, J. Sun, "Deep Residual Learning for Image Recognition", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778., 2016
[25] G. Huang, Z. Liu, L. Van Der Maaten and K. Q. Weinberger, "Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, doi: 10.1109/CVPR.2017.243., 2017
[26] G. S. Chung, C. S. Won, "Filter Pruning by Image Channel Reduction in Pre-trained Nonvolutional Neural Networks", Multimedia Tools and Applications, Springer, 2020
[27] J. Wang, T. Jiang, Z. Cui, Z. Cao, "Filter Pruning with a Feature Map Entropy Importance Criterion for Convolution Neural Networks Compressing", Elsevier, 2021
[28] S. Yeom, P. Seegerer, S. Lapuschkin, A. Binder, S. Wiedemann, K. Muller, W. Samek, "Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning", Pattern Recognition, Elsevier, 2021
[29] X. Sui, Q. Lv, L. Zhi, B. Zhu, Y. Yang, Y. Zhang, Z. Tan, "A Hardware-Friendly High-Precision CNN Pruning Method and Its FPGA Implementation", Sensors, MDPI, 2023
[30] Z. Wang, C. Li, X. Wang, "Convolutional Neural Network Pruning with Structural Redundancy Reduction", Computer Vision and Pattern Recognition, IEEE, 2021
[31] M. Lin, Y. Zhang, Y. Li, B. Chen, F. Chao, M. Wang, S. Li, Y. Tian, R. Ji, "1xN Pattern for Pruning Convolutional Neural Networks", Pattern Analysis and Machine Intelligence, IEEE Transactions, 2022.