Temporal assessment of cognitive load factors using ocular features during a visual search

Minoru Nakayama; Tomomi Okano

doi:10.4108/eetcasa.8797

Authors

Minoru Nakayama Institute of Science Tokyo
Tomomi Okano Institute of Science Tokyo

DOI:

https://doi.org/10.4108/eetcasa.8797

Keywords:

cognitive load, microsaccade rate, pupil change, modelling, chronological analysis

Abstract

The possibility of evaluating temporal changes in cognitive workloads during a visual search task is examined using microsaccade (MS) rates and pupillary changes. The experimental task was designed as a search for a speciﬁc ﬁgure, where task diﬃculty and reaction accuracy during the trials were controlled. Individual cognitive workloads were measured after the experimental sessions were conducted, using NASA-TLX scale ratings. Temporal changes in the cognitive load were identiﬁed using metrics of oculomotors during two stages of task processing, by comparing cognitive loads with individual ratings on a scale. Since the source of the load may be a common one, changes in latent attention resources required for the task were estimated with a designated state-space model, using the observation data in order to synthesise measurement of MS rates and pupillary changes. The predicted levels of attention resources correspond to the activity during the performance of the experimental tasks during the trials, and reﬂected some of the rating scores for workload scales. Also, the ranges of conﬁdence intervals for attention resources correlate signiﬁcantly with the ratings for information processing at the stage where visual stimulus is presented during tasks.

References

[1] Cairns, P. and Cox, A.L. (2008) Research Methods for Human-Computer Interaction (Cambridge, UK: Cam- bridge University Press).

[2] Preece, J., Rogers, Y. and Sharp, H. (2015) Interaction Design beyond human-computer interaction, 4th edition (Chichester, UK: John Wiley & Sons Ltd.).

[3] Kohama, T., Nakai, Y., Ohtani, S., Yamamoto, M., Ueda, S. and Yoshida, H. (2017) Quantitative comparison of cognitive load derived from voice or manual responses based on microsaccade rate analysis. Journal of Human Interface Society 19(2): 189–198.

[4] Dalmaso, M., Castelli, L., Scatturin, P. and Galfano, G. (2017) Working memory load modulates microsac- cade rate. Journal of Vision 17(3): 1–12.

[5] Krejtz, K., Duchowski, A.T., Niedzielska, A., Biele, C. and Krejtz, I. (2018) Eye tracking cognitive load using pupil diameter and microsaccades with ﬁxed gaze. PloS One 13: 1–23.

[6] Dalmaso, M., Castelli, L. and Galfano, G. (2020) Microsaccadic rate and pupil size dynamics in pro-/anti-saccade preparation: the impact of intermixed vs. blocked trial administration. Psychological Research 84: 1320–1332.

[7] Klein, C. and Ettinger, U. (2019) Eye Movement Research, An Introduction to its Scientiﬁc Foundations and Applications (Springer Nature Switzerland AG).

[8] Nakayama, R., Bardin, J.B., Koizumi, A., Motoyoshi, I. and Amano, K. (2022) Building a decoder of perceptual decisions from microsaccades and pupil size. Frontiers in Psychology 13: 942859.

[9] Okano, T. and Nakayama, M. (2021) Feasibility of evaluating temporal changes in cognitive load factors using ocular features. In COGAIN workshop, Proceedings of ETRA21: 29:1–6. https://doi.org/10.1145/3448018.3458019.

[10] Okano, T. and Nakayama, M. (2022) Research on time series evaluation of cognitive load factors using features of eye movement. In COGAIN workshop, Proceedings of ETRA22: 61:1–6. https://doi.org/10.1145/3448018.3458019.

[11] Dubiel, M., Nakayama, M. and Wang, X. (2023) Mod- elling attention levels with ocular responses in a speech- in-noise recall task. In Proceedings of ETRA2023 (ACM): 89:1–7. https://doi.org/10.1145/3588015.3589665.

[12] Nakayama, M. and Ueno, T. (2023) Estimation of latent attention resources using microsaccade frequency during a dual task. In Proceedings of ETRA2023 (ACM): 41:1–2. https://doi.org/10.1145/3588015.3590120.

[13] Nakayama, M. and Okano, T. (2024) Extracting cognitive workload factors on modelling attention resources using ocular information in ﬁgure search task. In Proceedings of 28th International Conference Information Visualisation iV: 95–100. https://doi.org/10.1109/IV64223.2024.00026.

[14] Hart, S.G. (2006) NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and Ergonomics Society 50th Annual meeting: 904–908.

[15] Miyake, S. (2015) Special issues no. 3: Measurement technique for ergonomics, section 3:psychological mea- surements and analyses (6), mental workload assess- ment and analysis -a reconsideration of the NASA- TLX- (Japanese). Ningen Kougaku (The Japanese journal of ergonomics) 51(6).

[16] Miyake, S. and Kuma-Shiro, M. (1993) Subjective mental workload assessment technique-an introduction to NASA-TLX and SWAT and a proposal of simple scoring methods- (Japanese). Ningen Kougaku (The Japanese journal of ergonomics) 29(6).

[17] Haruki Mizushina, Kiyomi Sakamoto, H.K. (2011) Relationship between psychological stress induced by workload and dynamic characteristics of saccadic eye movements during task execution. Transactions of the Institute of Electronics, Information and Communication Engineers Vol.j94-D No.10 pp.1640-1651 .

[18] Ziv, G. (2016) Gaze behavior and visual attention: A review of eye tracking studies in aviation. The International Journal of Aviation Psychology 26(3–4): 75– 104.

[19] Peiß, S., Wickens, C.D. and Baruah, R. (2018) Eye- tracking measures in aviation: A selective literature review. The International Journal of Aerospace Psychology 28(Issue 3–4): 98–112.

[20] Duchowski, A.T., Krejtz, K., Krejtz, I., Biele, C., Niedzielska, A., Kiefer, P., Raubal, M. et al. (2018) The index of pupillary activity: Measuring cognitive load vis- à-vis task diﬃculty with pupil oscillation.

[21] Engbert, R. and Kliegl, R. (2003) Microsaccades uncover the orientation of covert attention. Vision Research 43: 1035–1045.

[22] Kashihara, K., Okanoya, K. and Kawai, N. (2014) Emotional attention modulates microsaccadic rate and direction. Psychological Research 78: 166–179.

[23] Nakayama, M. and Hayakawa, Y. (2021) Inﬂuence of task evoked mental workloads on oculo-motor indices and their connections. EAI Trans. Context-aware Systems and Applications 7(23): e2:1–10.

[24] Wagenmakers, E.J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R. et al. (2018) Bayesian inference for psychology. part i: Theoretical advantages and practical ramiﬁcations. Psychonomic Bulletin & Review 25: 35–57.

[25] Dienes, Z. and Mclatchie, N. (2018) Four reasons to prefer Bayesian analyses over signiﬁcance testing. Psychonomic Bulletin & Review 25: 207–218.

[26] Nguyen, M.H., La, V.P., Le, T.T. and Vuong, Q.H. (2022) Introduction to Bayesian mindsponge framework analyt- ics: An innovative method for social and psychological research. MethodsX 9(101808): 1–16.

[27] Lee, M.D. (2011) How cognitive modeling can beneﬁt from hierarchical Bayesian models. Journal of Mathemat- ical Psychology 55: 1–7.

[28] Haaf, J.M. and Rounder, J.N. (2017) Developing constraint in Bayesian mixed models. Psychological Methods 22: 779–798.

[29] Schönbrodt, F.D. and Wagenmakers, E.J. (2018) Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bulletin & Review 25: 128–142.

[30] Muto, H. and Nagai, M. (2020) Mental rotation of cubes with a snake face: The role of the human-body analogy revisited. Visual Cognition 28: 106–111.

[31] Muto, H. (2021) Evidence for mixed processes in normal/mirror discrimination of rotated letters: A Bayesian model comparison between single- and mixed- distribution models. Japanese Psychological Research 63(3): 190–202.

[32] Nakayama, M. and Ueno, T. (2023) Latent attention resource estimation of peripheral visual stimuli using microsaccade frequency modellin. In Proceedings of 27th International Conference on Information Visualisation (iV) (IEEE): 142–147.

[33] OpenCV (2017), Staticsaliency. https://docs.opencv.org/4.x/d5/d87/ classcv_1_1saliency_1_1StaticSaliency.html.

[34] Engbert, R. (2006) Microsaccades: a microcosm for research on oculomotor control, attention, and visual perception. Progress in Brain Research 154: 177–192.

[35] Engbert, R., Sinn, P., Mergenthaler, K. and Truken- brod, H. (2015) Microsaccade toolbox 0.9. Potsdam Mind Research Repository .

[36] Haga, S. and Mizugami, N. (1996) Mental workload measurement using NASA-TLX (Japanese version) - sensitivity of workload scores to diﬃculty of various laboratory experiments-. Human Engineering 32(2), 71-79 .