Saliency Detection in Eye Gaze Prediction by Using Deep Learning

Document Type : Persian Original Article

Authors

1 University of Birjand

2 دانشگاه بیرجند

Abstract

Salient object detection attracted the attention of researchers in various fields, so that it is used in many applications of the visual machine, such as object detection and tracking. Most of the work in this area is based on bottom-top saliency models and use low-level features to extract the final saliency map that these works do not have a significant accuracy in salient object detection. On the other hand, top-down visual models are used for the specific applications. In this paper, a different method has been proposed to extract the salient object map that uses low-up and top-down attributes for extracting the salient and it is based on the learning process. The simultaneous selection of these features makes the proposed algorithm for various applications and increases the accuracy of the salient object detection. The learning process is performed by using the and Convolutional Neural Network (CNN) structure. After the decomposition of image to its super pixels, different features of image are extracted. Then, the extracted features are normalized to have zero mean and unit variance, and CNN is used to train the features. The accuracy of the proposed method is improved by using of the 8 salient and CNN. The performance of the proposed method has been compared to twenty method by applying four new databases including MSRA-100, ECSSD, MSRA-10K and Paskal-S. The proposed method provides better results compared to the other methods in respect to prediction of salient object.

Keywords


  [1]     A. Borji, D. Sihite, and L. Itti, “Quantitative analysis of humanmodel agreement in visual saliency modeling: A comparative study,” IEEE TIP, vol. 22, no. 1, pp. 55–69, 2013.
  [2]     A. Borji. Boosting bottom-up and top-down visual features for saliency estimation. In CVPR, 2012.
  [3]     Q. Zhou, “Object-based attention: saliency detection using contrast via background prototypes,” electronics letters, vol. 50, No. 14, pp. 997-999, 2014.
  [4]     L. Itti, C. Koch, and E. Niebur, “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, 1998.
  [5]     Chenlei Guo and Liming Zhang, “A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image and Video Compression,” IEEE Transactions On Image Processing, Vol. 19, No. 1, pp. 10-25, 2010.
  [6]     N. Sang, L. Wei, and Y. Wang, “A Biologically-inspired Top-down Learning Model Based on Visual Attention,” International conference on pattern recognition, 2010.
  [7]     Tadmeri Narayan Vikram, Marko Tscherepanow, Britta Wrede, “A saliency map based on sampling an image into random rectangular,” Pattern Recognition, 2012.
  [8]     M. Wang, J. Li, T. Huang, Y. Tian, L. Duan, G. Jia, Saliency detection based on 2d log-gabor wavelets and center bias, in: Proceedings of the International Conference on Multimedia, 2010, pp. 979–982.
  [9]     V. Yanulevskaya, J.M. Geusebroek, Significance of the Weibull distribution and its sub-models in natural image statistics, in: Proceedings of the International Conference on Computer Vision Theory and Applications, 2009, pp. 355–362.
[10]     R. Achanta and S. Susstrunk, “Saliency Detection for Contentaware Image Resizing,” in IEEE International Conference on Image Processing, 2009.
[11]     L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell, “SUN: A Bayesian framework for saliency using natural statistics,” Journal of Vision, vol. 8, no. 7, pp. 1–20, 2008.
[12]     T. Avraham, M. Lindenbaum, “Esaliency (extended saliency): meaningful attention using stochastic image modeling, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 4,  pp. 693–708, 2010.
[13]     W. Kienzle, F.A. Wichmann, B. Sch¨ olkopf, M.O. Franz, “A nonparametric approach to bottom-up visual saliency, Advances in Neural Information Processing Systems, 2007, pp. 689–696.
[14]     K. Huang, C. Zhu, and G. Li, “Saliency Detection by Adaptive Channel Fusion,” IEEE Signal Processing Letters, vol. 25, no. 7, pp. 1059–1063, 2018.
[15]     Q. Zhao and C. Koch, Learning visual saliency by combining feature maps in a nonlinear manner using adaboost", Journal of Vision, vol. 12, no. 6, pp. 1-15, 2012.
[16]     T. Judd, K. Ehinger, F. Durand, and A. Torralba, Learning to predict where humans look", in Proceedings of International Conference on Computer Vision, 2009.
[17]     Zhao and C. Koch, Learning a saliency map using _xated locations in natural scenes", Journal of Vision, vol. 11, no. n3, pp. 1-15, 2011.
[18]     B. Jiang, L. Zhang, H. Lu, C. Yang, and M.-H. Yang, “Saliency Detection via Absorbing Markov Chain,” 2013 IEEE International Conference on Computer Vision, Dec. 2013.
[19]     N. Tong, H. Lu, L. Zhang, X. Ruan, "Saliency Detection with Multi-Scale Superpixels", IEEE Signal Processing Letters, Vol. 21, No. 9, pp. 12-19, 2014.
[20]     Wang, H. Lu, X. Li, N. Tong, and W. Liu, “Saliency detection via background and foreground seed selection,” Neurocomputing, vol. 152, pp. 359–368, Mar. 2015.
[21]     C. Aytekin, H. Possegger, T. Mauthner, S. Kiranyaz, H. Bischof, and M. Gabbouj, “Spatiotemporal Saliency Estimation by Spectral Foreground Detection,” IEEE Transactions on Multimedia, vol. 20, no. 1, pp. 82–95, Jan. 2018.
[22]     J. Yang and M.-H. Yang, “Top-Down Visual Saliency via Joint CRF and Dictionary Learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 3, pp. 576–588, Mar. 2017.
[23]     H. Farsi, R. Nasiripour, S. Mohammadzadeh, "Eye gaze detection based on learning automata by using SURF descriptor," Information Systems & Telecommunication, vol. 6, no. 1, pp. 41-49, 2018.
[24]     P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient graph-based image segmentation,” IJCV, vol. 59, no. 2, pp. 167–181, 2004.
[25]     O., Csillik, "Fast segmentation and classification of very high resolution remote sensing data using SLIC superpixels," Remote Sensing, vol. 9, no. 3, pp. 243-250, 2017.
[26]     N. Tong, Lu H, R. Xiang, M. Hsuan. "Salient object detection via bootstrap learning", IEEE Conference on Computer Vision and Pattern Recognition, USA, pp. 1884-1892, 2015.
[27]     G., Li, Y., Yu, "Deep contrast learning for salient object detection," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 478-487, 2016.
[28]     E. A. Smirnov, D. M. Timoshenko, and S. N. Andrianov, "Comparison of Regularization Methods for ImageNet Classification with Deep Convolutional Neural Networks," AASRI Procedia, vol. 6, 2014, pp. 89–94.
[29]     R. Achanta, S. S. Hemami, F. J. Estrada, and S. S€usstrunk, “Frequency-tuned salient region detection,” in Proc. IEEE Conf. Computer. Vis. Pattern Recognition, 2009, pp. 1597–160.
[30]     Yan Q, Xu L, Shi J, Jiaya Jia, "Hierarchical saliency detection", IEEE Conference, USA, 2013, pp. 1155-1162.
[31]     http://mmcheng.net/gsal/.
[32]     Y. Li, X. Hou, C. Koch, J. Rehg, and A. Yuille, “The secrets of salient object segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 280–287.
[33]     Cheng M, Mitra NJ, Huang X, Torr PH, Hu S. Global contrast based salient region detection. Pattern Analysis and Machine Intelligence, IEEE Transactions on. 2015; 37(3):569–82.
[34]     F. Perazzi, P. Kr€ahenb€uhl, Y. Pritch, and A. Hornung, “Saliency filters: Contrast based filtering for salient region detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 733–740.
[35]     X. Li, Y. Li, C. Shen, A. R. Dick, and A. van den Hengel,“Contextual hypergraph modeling for salient object detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3328–3335.
[36]     X. Shen and Y. Wu, “A unified approach to salient object detection via low rank matrix recovery,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2296–2303.
[37]     C. Lang, G. Liu, J. Yu, and S. Yan, “Saliency detection by multitask sparsity pursuit,” IEEE Trans. Image Process., vol. 21, no. 3, pp. 1327–1338, 2012.
[38]     H. Jiang, J. Wang, Z. Yuan, T. Liu, N. Zheng, and S. Li, “Automatic salient object segmentation based on context and shape prior,” in Proc. Brit. Mach. Vis. Conf., 2011, pp. 1–12.
[39]     S. Goferman, L. Z. Manor, and A. Tal, “Context-aware saliency detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 1915–1926.
[40]     X. Hou, J. Harel, and C. Koch, “Image signature: Highlighting sparse salient regions,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp. 194–201, 2012.
[41]     E. Rahtu, J. Kannala, M. Salo, and J. Heikkil€a, “Segmenting salient objects from images and videos,” in Proc. Eur. Conf. Comput. Vis., 2010, pp. 366–379.
[42]     R. Achanta, S. Hemami, F. Estrada, and S. S€usstrunk, “Frequencytuned salient region detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1597–1604.
[43]     X. Hou and L. Zhang, “Saliency detection: A spectral residual approach,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1–8.
[44]     Y. Zhai and M. Shah, “Visual attention detection in video sequences using spatiotemporal cues,” in Proc. 14th ACM Int. Conf. Multimedia, 2006, pp. 815–824.
[45]     J. Kim, D. Han, Y. Tai, and J. Kim, “Salient region detection via high-dimensional color transform,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 883–890.
[46]     R. Margolin, A. Tal, and L. Zelnik-Manor, “What makes a patch distinct?” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1139–1146.
[47]     C. Scharfenberger, A. Wong, K. Fergani, J. S. Zelek, and D. A. Clausi, “Statistical textural distinctiveness for salient region detection in natural images,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 979–986.
[48]     K.-Y. Chang, T.-L. Liu, H.-T. Chen, and S.-H. Lai, “Fusing generic objectness and visual saliency for salient object detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 914–921.
[49]     R. Achanta, F. J. Estrada, P. Wils, and S. S€usstrunk, “Salient region detection and segmentation,” in Proc. 6th Int. Conf. Comput. Vis. Syst., 2008, pp. 66–75.
[50]     N.D.B. Bruce, J.K. Tsotsos, Saliency based on information maximization, Advances in Neural Information Processing Systems, 2005, pp. 155–162.
[51]     J. Harel, C. Koch, P. Perona, Graph-based visual saliency, Advances in Neural Information Processing Systems, 2007, pp. 545–552.
[52]     Y. F. Ma and H. J. Zhang, Contrast-based image attention analysis by using fuzzy growing. In ACM International Conference on Multimedia, 2003.
[53]     X. Hou and L. Zhang. Saliency detection: A spectral residual approach. IEEE Conference on Computer Vision and Pattern Recognition, 2007.
[54]     A. Garcia-Diaz, V. Lebor´an, X. R. Fdez-Vidal, and X. M. Pardo. On the relationship between optical variability, visual saliency, and eye fixations: A computational approach. Journal of Vision, Vol. 12, No. 6, 2012.
[55]     X. Hou and L. Zhang. Dynamic visual attention: Searching for coding length increments. In NIPS, pages 681–688, 2008.
[56]     B., Jiang, L., Zhang, H., Lu, Saliency detection via absorbing Markov chain. IEEE Int. Conf. on Computer Vision, Sydney, Australia, 2013.
[57]     X., Hu, L., Zhu, J., Qin, C.W., Fu, P.A. Heng. Recurrently aggregating deep features for salient object detection. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[58]     X., Zhang, T., Wang, J., Qi, H., Lu, G. Wang. Progressive attention guided recurrent network for salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 714-722.
[59]     W., Wang, J., Shen, X., Dong, A. Borji. Salient object detection driven by fixation prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1711-1720.
[60]     S., Fang, J., Li, Y., Tian, T., Huang, X., Chen, Learning discriminative subspaces on random contrasts for image saliency analysis. IEEE transactions on neural networks and learning systems, vol. 28, no. 5, pp.1095-1108, 2018.
[61]     B., Ghariba, M.S., Shehata, P. McGuire. Visual Saliency Prediction Based on Deep Learning. Information, vol. 10, no. 8, 2019.