On the Consistency of 360 Video Quality Assessment in Repeated Subjective Tests: A Pilot Study

Majed Elwardy; Hans-Juergen Zepernick; Thi My Chinh Chu; Yan Hu

doi:10.4108/eetinis.v11i1.4323

Authors

Majed Elwardy Blekinge Institute of Technology
Hans-Juergen Zepernick Blekinge Institute of Technology
Thi My Chinh Chu Blekinge Institute of Technology
Yan Hu Blekinge Institute of Technology

DOI:

https://doi.org/10.4108/eetinis.v11i1.4323

Keywords:

360 video, subjective tests, quality of experience, quality assessment, pilot study, annotated dataset, opportunity-limited conditions, standing viewing, seated viewing

Abstract

Immersive media such as virtual reality, augmented reality, and 360◦ video have seen tremendous technological developments in recent years. Furthermore, the advances in head-mounted displays (HMDs) offer the users increased immersive experiences compared to conventional displays. To develop novel immersive media systems and services that satisfy the expectations of the users, it is essential to conduct subjective tests revealing users’ perceived quality of immersive media. However, due to the new viewing dimensions provided by HMDs and the potential of interacting with the content, a wide range of subjective tests are required to understand the many aspects of user behavior in and quality perception of immersive media. The ground truth obtained by such subjective tests enable the development of optimized immersive media systems that fulfill the expectations of the users. This article focuses on the consistency of 360◦ video quality assessment to reveal whether users’ subjective quality assessment of such immersive visual stimuli changes fundamentally over time or is kept consistent with each user having their own behavior signature. A pilot study was conducted under pandemic conditions with participants given the task of rating the quality of 360◦ video stimuli on an HMD in standing and seated viewing. The choice of conducting a pilot study is motivated by the fact that immersive media impose high cognitive load on the participants and the need to keep the number of participants under pandemic conditions as low as possible. To gain insight into the consistency of the participants’ 360◦ video assessment over time, three sessions were held for each participant and each viewing condition with long and short breaks between sessions. In particular, the opinion scores and head movements were recorded for each participant and each session in standing and seated viewing. The statistical analysis of this data leads to the conjecture that the quality rating stays consistent throughout these sessions with each participant having their own quality assessment signature. The head movements, indicating the participants’ scene exploration during the quality assessment task, also remain consistent for each participant according their individual narrower or wider scene exploration signature. These findings are more pronounced for standing viewing than for seated viewing. This work supports the role of pilot studies being a useful approach of conducting pre-tests on immersive media quality under opportunity-limited conditions and for the planning of subsequent full subjective tests with a large panel of participants. The annotated RQA360 dataset containing the data recorded in the repeated subjective tests is made publicly available to the research community.

Downloads

Download data is not yet available.

References

Aloqaily, M., Bouachir, O., Karray, F., Al Ridhawi, I. and Saddik, A.E. (2023) Integrating Digital Twin and Advanced Intelligent Technologies to Realize the Metaverse. IEEE Consumer Electronics Magazine 12(6): 47–55. DOI: https://doi.org/10.1109/MCE.2022.3212570

Han, Y., Niyato, D., Leung, C., Miao, C. and Kim, D.I. (2022) A Dynamic Resource Allocation Framework for Synchronizing Metaverse with IoT Service and Data. In Proc. IEEE Int. Conf. on Communications (Seoul, Republic of Korea): 298–301. DOI: https://doi.org/10.1109/ICC45855.2022.9838422

3GPP TR 26.918 V16.0.0 (2018) Virtual Reality (VR) Media Services Over 3GPP (Release 16), 3rd Generation Partnership Project, Technical Specification Group Services and System Aspects.

Yrjölä, S., Ahokangas, P. and Matinmikko-Blue, M. (Jun. 2020) White Paper on Business of 6G. (6G Research Visions, No. 3), University of Oulu, Finland.

Duanmu, Z., Zeng, K., Ma, K., Rehman, A. and Wang, Z. (2017) A Quality-of-Experience Index for Streaming Video. IEEE Journal of Selected Topics in Signal Processing 11(1): 154–166. DOI: https://doi.org/10.1109/JSTSP.2016.2608329

Hu, Y., Elwardy, M. and Zepernick H.-J. (2021) On the Effect of Standing and Seated Viewing of 360◦ Videos on Subjective Quality Assessment: A Pilot Study. Computers 10(6): 1–28. DOI: https://doi.org/10.3390/computers10060080

Liu, R., Peng, C., Zhang, Y., Husarek, H. and Yu, Q. (2021) A Survey of Immersive Technologies and Applications for Industrial Product Development. Computers & Graphics 100: 137–151. DOI: https://doi.org/10.1016/j.cag.2021.07.023

Cunningham, D.W. and Wallraven, C. (2012) Experimental Design: From User Studies to Psychophysics (Boca Raton, FL: CRC Press).

Recommendation ITU-R BT.500-13 (2012) Methodology for the Subjective Assessment of the Quality of Television Pictures, International Telecommunication Union, Geneva, Switzerland.

Recommendation ITU-T BT.1788 (2007) Methodology for the Subjective Assessment of Video Quality in Multimedia Applications, International Telecommunication Union, Geneva, Switzerland.

Recommendation ITU-T P.910 (2008) Subjective Video Quality Assessment Methods for Multimedia Applications, International Telecommunication Union, Geneva, Switzerland.

Recommendation ITU-T P.915 (2016) Subjective Assessment Methods for 3D Video Quality, International Telecommunication Union, Geneva, Switzerland.

Recommendation ITU-T P.919 (2020) Subjective Test Methodologies for 360◦ Video on Head-Mounted Displays, International Telecommunication Union, Geneva, Switzerland.

Recommendation ITU-T P.1320 (2022) Quality of Experience Assessment of Extended Reality Meetings, International Telecommunication Union, Geneva, Switzerland.

J. Gutierrez, et al. (2022) Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360 Videos: ITU-T Rec. P.919. IEEE Trans. Multimedia 24: 3087–3100. DOI: https://doi.org/10.1109/TMM.2021.3093717

Feil-Seifer, D., Haring, K.S., Rossi, S., Wagner, A.R. and Williams, T. (2020) Where to Next? The Impact of COVID-19 on Human-Robot Interaction Research. ACM Trans. Human-Robot Interaction 10(1): 1–7. DOI: https://doi.org/10.1145/3405450

A. Steed, et al. (2020) Evaluating Immersive Experiences During Covid-19 and Beyond. Interaction 27(4): 62–67. DOI: https://doi.org/10.1145/3406098

Spang, R.P. and Pieper, K. (2021) Durchführung von psychophysiologischen und subjektiven Experimenten während einer Pandemie. ITG News (2): 13–14.

Zepernick H.-J., Pieper, K., Spang, R.P., Engelke, U., Hirth, M. and Naderi, B. (2021) On the Impact of COVID-19 on Subjective Digital Media Quality Assessment. In Proc. IEEE Int. Workshop on Multimedia Signal Processing (Tampere, Finland): 1–6. DOI: https://doi.org/10.1109/MMSP53017.2021.9733583

Perez, P., Janowski, L., Garcia, N. and Pinson, M. (2022) Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR). IEEE Trans. Multimedia 24: 3442–3454. DOI: https://doi.org/10.1109/TMM.2021.3098450

Elwardy, M., Zepernick H.-J. and Hu, Y. (2021) On Head Movements in Repeated 360◦ Video Quality Assessment for Standing and Seated Viewing on Head Mounted Displays. In Proc. IEEE Conf. on Virtual Reality and 3D User Interfaces Abstracts and Workshops (Lisbon, Portugal): 71–74. DOI: https://doi.org/10.1109/VRW52623.2021.00020

Elwardy, M., Zepernick H.-J., Chu, T.M.C. and Hu, Y. (2021) On the Opinion Score Consistency in Repeated 360◦ Video Quality Assessment for Standing and Seated Viewing on Head-Mounted Displays. In Proc. IEEE Int. Conf. on Signal Processing and Commun. Systems (Sydney, Australia): 1–10. DOI: https://doi.org/10.1109/ICSPCS53099.2021.9660331

Elwardy, M., Zepernick H.-J., Sundstedt, V. and Hu, Y. (2019) Impact of Participants’ Experiences with Immersive Multimedia on 360◦ Video Quality Assessment. In Proc. IEEE Int. Conf. on Signal Processing and Commun. Systems (Gold Coast, Australia): 40–49. DOI: https://doi.org/10.1109/ICSPCS47537.2019.9008739

Beihang University, School of Electronic and Information Engineering, Beijing, China VQA-ODV. 2017, Accessed on: 12. Sep. 2020. URL https://github.com/ Archer-Tatsu/VQA-ODV.

Li, C., Xu, M. and Wang, Z. (2018) Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video: A Large-Scale Dataset and a Deep Learning Model. In Proc. ACM Int. Conf. on Multimedia (Seoul, Republic of Korea): 932–940. DOI: https://doi.org/10.1145/3240508.3240581

Zhang, Y., Wang, Y., Liu, F., Liu, Z., Li, Y., Yang, D. and Chen, Z. (2018) Subjective Panoramic Video Quality Assessment Database for Coding Applications. IEEE Trans. Broadcast. 64(2): 42–51. DOI: https://doi.org/10.1109/TBC.2018.2811627

FFmpeg H.264 Video Encoding Guide. 2018, Accessed on: 12. Sep. 2020. URL https://trac.ffmpeg.org/wiki/Encode/H.264#crf.

FFmpeg FFmpeg and H.265 Encoding Guide. 2018, Accessed on: 12 Sep. 2020. URL https://trac.ffmpeg.org/wiki/Encode/H.265.

Blekinge Institute of Technology, Karlskrona, Sweden RQA360. 2023, Accessed on: 28. Aug. 2023. URL https://github.com/MajedElwardy/RQA360.

Dodge, Y. (2008) The Concise Encyclopedia of Statistics (New York, NY, USA: Springer).

Telab, N.N. What Scientific Idea is Ready for Retirement?, Accessed on: 17. Jul. 2023. URL https://web.archive.org/web/20140116031136/http://www.edge.org/response-detail/25401.

The MathWorks Inc. (2023) Statistics and Machine Learning Toolbox (R2021b), Natick, MA, USA. URL https://www.mathworks.com.

Westfall, P.H. (2014) Kurtosis as Peakedness, 1905- 2014. R.I.P. The American Statistical Association 68(3): 191–195. DOI: https://doi.org/10.1080/00031305.2014.917055

Neter, J., Kutner, M.H., Nachtsheim, C.J. and Wasserman, W. (1996) Applied Linear Statistical Models (New York, NY, USA: McGraw-Hill), 5th ed.

Wu, C.F.J. and Hamada, M. (2000) Experiments: Planning, Analysis, and Parameter Design Optimization (New York, NY, USA: John Wiley & Sons).

Elwardy, M., Hu, Y., Zepernick H.-J., Chu, T.M.C. and Sundstedt, V. (2020) Comparison of ACR Methods for 360◦ Video Quality Assessment Subject to Participants’ Experience with Immersive Media. In Proc. Int. Conf. on Signal Process. and Commun. Systems (Adelaide, Australia): 1–10. DOI: https://doi.org/10.1109/ICSPCS50536.2020.9310071

Blekinge Institute of Technology Visual and Interactive Computing Laboratory (ViaLab). URL https://a.bth.se/viatech-synergy/vialabs/.

Brunnström, K., Andrén, B., Schenkman, B., Djupsjöbacka, A. and Hamsis, O. (2020) Recommended Precautions Because of Covid-19 for Perceptual, Behavioural, Quality and User Experiments with Test Persons in Indoor Labs, RISE Report:2020:84, Stockholm, Sweden.