Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

How saccadic models help predict where we look during a visual task? Application to visual quality assessment.

498 visualizaciones

Publicado el

We present saccadic models which are an alternative way to predict where observers look at. Compared to saliency models, saccadic models generate plausible visual scanpaths from which saliency maps can be computed. In addition these models have the advantage of being adaptable to different viewing conditions, viewing tasks and types of visual scene. We demonstrate that saccadic models perform better than existing saliency models for predicting where an observer looks at in free-viewing condition and quality-task condition (i.e. when observers have to score the quality of an image). For that, the joint distributions of saccade amplitudes and orientations in both conditions (i.e. free-viewing and quality task) have been estimated from eye tracking data. Thanks to saccadic models, we hope we will be able to improve upon the performance of saliency-based quality metrics, and more generally the capacity to predict where we look within visual scenes when performing visual tasks.

Publicado en: Ciencias
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

How saccadic models help predict where we look during a visual task? Application to visual quality assessment.

  1. 1. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion How saccadic models help predict where we look during a visual task? Application to visual quality assessment. Olivier Le Meur 1 Antoine Coutrot 2 olemeur@irisa.fr 1 IRISA - University of Rennes 1, France 2 CoMPLEX, University College London, UK February 17, 2016 1 / 25
  2. 2. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Preamble These slides are based on the following papers: O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015, doi:10.1016/j.visres.2014.12.026. O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. 2 / 25
  3. 3. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Outline 1 Introduction 2 Saliency models 3 Saccadic models 4 Tuning saccadic model 5 Conclusion 3 / 25
  4. 4. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Outline 1 Introduction Covert vs overt Bottom-Up vs Top-Down 2 Saliency models 3 Saccadic models 4 Tuning saccadic model 5 Conclusion 4 / 25
  5. 5. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Introduction (1/2) Visual attention Posner proposed the following definition (Posner, 1980). Visual atten- tion is used: ª to select important areas of our visual field (alerting); ª to search for a target in cluttered scenes (searching). There are several kinds of visual attention: ª Overt visual attention: involving eye movements; ª Covert visual attention: without eye movements (Covert fixations are not observable). 5 / 25
  6. 6. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Introduction (2/2) Bottom-Up vs Top-Down ª Bottom-Up: some things draw attention reflexively, in a task-independent way (Involuntary; Very quick; Unconscious); ª Top-Down: some things draw volitional attention, in a task-dependent way (Voluntary; Very slow; Conscious). 6 / 25
  7. 7. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Outline 1 Introduction 2 Saliency models 3 Saccadic models 4 Tuning saccadic model 5 Conclusion 7 / 25
  8. 8. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saliency model (1/3) Computational models of visual attention aim at predicting where we look within a scene. ª Saliency model computes a 2D static saliency map from an input image. ª Most of state-of-the-art models simulate the Bottom-Up Overt visual attention. 8 / 25
  9. 9. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saliency model (2/3) Taxonomy of models: ª Information Theoretic models; ª Cognitive models; ª Graphical models; ª Spectral analysis models; ª Pattern classification models; ª Bayesian models. Extracted from (Borji and Itti, 2013). 9 / 25
  10. 10. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saliency model (3/3) The picture is much clearer than 10 years ago! New datasets, new metrics/benchmarks, new protocols, new methods, new devices..... 10 / 25
  11. 11. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saliency model (3/3) The picture is much clearer than 10 years ago! New datasets, new metrics/benchmarks, new protocols, new methods, new devices..... BUT Important aspects of our visual system are clearly overlooked: Current models implicitly assume that eyes are equally likely to move in any direction; Viewing biases are not taken into account; The temporal dimension is not considered (static saliency map). 10 / 25
  12. 12. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Outline 1 Introduction 2 Saliency models 3 Saccadic models Presentation Proposed model 4 Tuning saccadic model 5 Conclusion 11 / 25
  13. 13. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Presentation (1/1) ª Eye movements are composed of fixations and saccades. A sequence of fixations is called a visual scanpath. ª When looking at visual scenes, we perform in average 4 visual fixations per second. Saccadic models are used: 1 to compute plausible visual scanpaths (stochastic, saccade amplitudes / orientations...); 2 to infer the scanpath-based saliency map ⇔ to predict salient areas!! 12 / 25
  14. 14. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (1/6) Let I : Ω ⊂ R2 → R3 an image and xt a fixation point at time t. We consider the 2D discrete conditional probability: p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU : Ω → [0, 1] is the grayscale saliency map; ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d is the saccade amplitude between two fixation points xt and xt−1 (expressed in degree of visual angle), and φ is the angle (expressed in degree between these two points); ª pM (x, t) represents the memory state of the location x at time t. This time-dependent term simulates the inhibition of return. O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015. O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. 13 / 25
  15. 15. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (1/6) Let I : Ω ⊂ R2 → R3 an image and xt a fixation point at time t. We consider the 2D discrete conditional probability: p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU : Ω → [0, 1] is the grayscale saliency map; ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d is the saccade amplitude between two fixation points xt and xt−1 (expressed in degree of visual angle), and φ is the angle (expressed in degree between these two points); ª pM (x, t) represents the memory state of the location x at time t. This time-dependent term simulates the inhibition of return. O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015. O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. 13 / 25
  16. 16. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (1/6) Let I : Ω ⊂ R2 → R3 an image and xt a fixation point at time t. We consider the 2D discrete conditional probability: p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU : Ω → [0, 1] is the grayscale saliency map; ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d is the saccade amplitude between two fixation points xt and xt−1 (expressed in degree of visual angle), and φ is the angle (expressed in degree between these two points); ª pM (x, t) represents the memory state of the location x at time t. This time-dependent term simulates the inhibition of return. O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015. O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. 13 / 25
  17. 17. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (1/6) Let I : Ω ⊂ R2 → R3 an image and xt a fixation point at time t. We consider the 2D discrete conditional probability: p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU : Ω → [0, 1] is the grayscale saliency map; ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d is the saccade amplitude between two fixation points xt and xt−1 (expressed in degree of visual angle), and φ is the angle (expressed in degree between these two points); ª pM (x, t) represents the memory state of the location x at time t. This time-dependent term simulates the inhibition of return. O. Le Meur & Z. Liu, Saccadic model of eye movements for free-viewing condition, Vision Research, 2015. O. Le Meur & A. Coutrot, Introducing context-dependent and spatially-variant viewing biases in saccadic models, Vision Research, 2016. 13 / 25
  18. 18. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (2/6) Bottom-up saliency map p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pBU is the bottom-up saliency map. • Saliency aggregation (Le Meur and Liu, 2014) GBVS model (Harel et al., 2006) and RARE2012 model (Riche et al., 2013). These models present a good trade-off between quality and complexity. • pBU (x) is constant over time. (Tatler et al., 2005) indeed demonstrated that bottom-up influences do not vanish over time. 14 / 25
  19. 19. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (3/6) Memory effect and inhibition of return (IoR) p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pM (x, t) represents the memory effect and IoR of the location x at time t. It is composed of two terms: Inhibition and Recovery. • The spatial IoR effect declines as a Gaussian function Φσi (d) with the Euclidean distance d from the attended location (Bennett and Pratt, 2001); • The temporal decline of the IoR effect is simulated by a simple linear model. 15 / 25
  20. 20. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Saccadic model (4/6) Viewing biases p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª pB(d, φ) represents the joint probability distribution of saccade amplitudes and orientations. d and φ represent the distance and the angle between each pair of successive fixations, respectively. 16 / 25
  21. 21. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Proposed model (5/6) Selecting the next fixation point p (x|xt−1, . . . , xt−T ) ∝ pBU (x)pB(d, φ)pM (x, t) ª Stochastic behavior: We generate Nc = 5 random locations according to the 2D discrete conditional probability p (x|xt−1, · · · , xt−T ). The location with the highest saliency gain is chosen as the next fixation point x∗ t . 17 / 25
  22. 22. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Proposed model (6/6) Scanpaths and saliency maps ª Scanpaths composed of 10 fixations: ª Scanpath-based saliency maps: (a) original image; (b) human saliency map; (c) GBVS saliency map; (d) GBVS-SM saliency maps computed from the simulated scanpaths. See papers for more details 18 / 25
  23. 23. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Outline 1 Introduction 2 Saliency models 3 Saccadic models 4 Tuning saccadic model A flexible framework Example: the quality task 5 Conclusion 19 / 25
  24. 24. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Tuning saccadic model (1/1) The proposed framework allows us to adapt the saccadic model for different conditions (thanks to the 2D conditional probability pM ): ª for specific visual scenes: scene-dependent and spatially-variant distributions. ª for specific category of people, e.g. young or elder. ª for a given task... 20 / 25
  25. 25. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Tuning saccadic model for a quality task (1/3) Our goal is to reproduce the visual behavior of observers while they are performing a quality task. We re-analyse eye tracking data of (Ninassi et al., 2007): ª FT: eye movements were recorded while observers freely looked at unimpaired images; ª Ref. QT: eye movements were recorded while observers looked at the unimpaired images during the quality task; ª Deg. QT: eye movements were recorded while observers looked at the degraded images during the quality task. Double Stimulus Impairment Scale was used to perform the subjective quality evaluation. 21 / 25
  26. 26. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Tuning saccadic model for a quality task (2/3) From left to right: FT, Ref. QT and Deg. QT. ª The joint distribution of the condition Deg. QT is less focussed than the two others ⇒ higher visual exploration. ª Observers performed smaller saccades in the condition Ref. QT compared to FT and Deg. QT conditions ⇒ Memorization. 22 / 25
  27. 27. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Tuning saccadic model for a quality task (3/3) ª Preliminary results: 23 / 25
  28. 28. Visual attention O. Le Meur Introduction Covert vs overt Bottom-Up vs Top-Down Saliency models Saccadic models Presentation Proposed model Tuning saccadic model A flexible framework Example: the quality task Conclusion Conclusion (1/1) The key take-home messages: ª A new saccadic model performing well to • produce plausible scanpaths; • detect the most salient regions of visual scenes. ª A flexible approach! Future works: ª Dealing with the limitations of the current implementation. The presentation is available on my slideShare account: http://fr.slideshare.net/OlivierLeMeur. 24 / 25
  29. 29. Thanks to Adrien.
  30. 30. Visual attention O. Le Meur References References P. J. Bennett and J. Pratt. The spatial distribution of inhibition of return:. Psychological Science, 12:76–80, 2001. A. Borji and L. Itti. State-of-the-art in visual attention modeling. IEEE Trans. on Pattern Analysis and Machine Intelligence, 35: 185–207, 2013. J. Harel, C. Koch, and P. Perona. Graph-based visual saliency. In Proceedings of Neural Information Processing Systems (NIPS), 2006. O. Le Meur and Z. Liu. Saliency aggregation: Does unity make strength? In ACCV, 2014. A. Ninassi, O. Le Meur, P. Le Callet, and D. Barba. Does where you gaze on an image affect your perception of quality? applying visual attention to image quality metric. In ICIP, 2007. M. I. Posner. Orienting of attention. Quarterly Journal of Experimental Psychology, 32:3–25, 1980. N. Riche, M. Mancas, M. Duvinage, M. Mibulumukini, B. Gosselin, and T. Dutoit. Rare2012: A multi-scale rarity-based saliency detection with its comparative statistical analysis. Signal Processing: Image Communication, 28(6):642 – 658, 2013. ISSN 0923-5965. doi: http://dx.doi.org/10.1016/j.image.2013.03.009. B.W. Tatler, R. J. Baddeley, and I.D. Gilchrist. Visual correlates of fixation selection: effects of scale and time. Vision Research, 45:643–659, 2005. 25 / 25

×