Crowdsourcing Remote Co-design Towards Improving the Validity and Reliability of mHealth Application Development – A Case Study on Sleep Solved: A mHealth App Designed Virtually with Teens
DOI:
https://doi.org/10.4108/eetpht.11.9416Keywords:
mHealth, digital health citizens, co-design, crowdsourcing, user experience, person-based design, sleep, teensAbstract
INTRODUCTION: Co-design has become a fundamental pillar of formative digital health research. Typically, this approach involves in–person workshops that involve a rich but limited amount of data. Virtually crowdsourcing co-design, however, provides the promise of rapid and vastly increased data. This is a novel, exploratory approach in mHealth design that may appease common health research concerns surrounding reliability and validity, whilst providing swifter feedback to meet product development timelines.
OBJECTIVES: The objective of this exploratory single case study was to explore the virtual, crowdsourced, co-design of Sleep Solved, an educational mHealth sleep app designed with teens. In doing so, we wished to learn which virtual methods were used to engage teens in the co-design and to explore how these virtual co-design methods can be adapted for large-scale ideation and testing.
METHODS: We conducted an enquiry-based iterative case study utilising the Bayazit 3-stage model. 85 teens participated over 11 months. Data was thematically analysed over several design iterations.
RESULTS: Rapid virtual feedback allowed for quick pivots in a short time frame. Four stages of feedback from teens led to iterative changes to scientific information contextualisation and user experience, from lo-fidelity mock-ups through to a coded app beta.
CONCLUSION: The co-design of Sleep Solved exemplified the potential of virtually crowdsourcing teens in mHealth. Key to this evolution will be the ability to leverage big data utilising AI and machine learning approaches to data collation and synthesization, such that meaningful and contextual findings can be applied in line with software development timelines.
Downloads
References
[1] Carmona NE, Usyatynsky A, Kutana S, et al (2021) A Transdiagnostic Self-management Web-Based App for Sleep Disturbance in Adolescents and Young Adults: Feasibility and Acceptability Study. JMIR Form Res 5:e25392
[2] Werner-Seidler A, Wong Q, Johnston L, et al (2019) Pilot evaluation of the Sleep Ninja: a smartphone application for adolescent insomnia symptoms. BMJ Open 9:e026502
[3] Ali N, Rigney G, Weiss SK, et al (2018) Optimizing an eHealth insomnia intervention for children with neurodevelopmental disorders: a Delphi study. Sleep Health 4:224–234
[4] Neher T, Green J, Puzia M, Huberty J (2022) Describing the use of a mindfulness-based app for sleep and mental well-being, across age, in children. Child Youth Care Forum 51:749–768
[5] Ahuja M (2018) Perceived credibility in mHealth apps : a case study on a sleep scheduling app for insomnia. University of Twente
[6] Alami H, Gagnon M-P, Fortin J-P (2017) Digital health and the challenge of health systems transformation. Mhealth 3:31
[7] Rowland SP, Fitzgerald JE, Holme T, et al (2020) What is the clinical value of mHealth for patients? NPJ Digit Med 3:4
[8] (2023) E-Health Market Analysis. In: Grandview Research. http://www.grandviewresearch.com/industry-analysis/e-health-market. Accessed 12 Oct 2023
[9] (2020) Global digital health market report 2020: market is expected to witness a 37.1% spike in growth in 2021 and will continue to grow and reach USD 508.8 billion by 2027. In: Globe Newswire. https://www.globenewswire.com/news-release/2020/11/25/2133473/0/en/Global-Digital-Health-Market-Report-2020-Market-is-Expected-to-Witness-a-37-1-Spike-in-Growth-in-2021-and-will-Continue-to-Grow-and-Reach-US-508-8-Billion-by-2027.html. Accessed 25 Sep 2023
[10] Ospina-Pinillos L, Davenport T, Mendoza Diaz A, et al (2019) Using Participatory Design Methodologies to Co-Design and Culturally Adapt the Spanish Version of the Mental Health eClinic: Qualitative Study. J Med Internet Res 21:e14127
[11] Hetrick SE, Robinson J, Burge E, et al (2018) Youth Codesign of a Mobile Phone App to Facilitate Self-Monitoring and Management of Mood Symptoms in Young People With Major Depression, Suicidal Ideation, and Self-Harm. JMIR Ment Health 5:e9
[12] Roberts C, Freeman J, Samdal O, et al (2009) The Health Behaviour in School-aged Children (HBSC) study: methodological developments and current tensions. Int J Public Health 54 Suppl 2:140–150
[13] Mugford J (2019) Development and usability testing of an eHealth sleep intervention program for Youth. Mount Saint Vincent University
[14] Cherenshchykova A, Miller AD (2021) Sociotechnical Design Opportunities for Pervasive Family Sleep Technologies. In: Proceedings of the 14th EAI International Conference on Pervasive Computing Technologies for Healthcare. Association for Computing Machinery, New York, NY, USA, pp 11–20
[15] Tougas M (2021) A user-centered approach in the development of an eHealth sleep intervention for adolescents with and without recurrent pain
[16] Bennett SE, Johnston MH, Treneman-Evans G, et al (2024) Using the person-based approach to co-create and optimize an app-based intervention to support better sleep for adolescents in the United Kingdom: Mixed methods study. JMIR Hum Factors 11:e63341
[17] Shriane AE, Vincent GE, Ferguson SA, et al (2025) Co-designed resources to improve sleep health in young shiftworkers: a qualitative study. Sleep Med 106511
[18] Pulantara IW, Parmanto B, Germain A (2018) Development of a Just-in-Time Adaptive mHealth Intervention for Insomnia: Usability Study. JMIR Hum Factors 5:e21
[19] Zhu H, Xiao L, Tu A (2022) Effectiveness of technology-based interventions for improving sleep among children: a systematic review and meta-analysis. Sleep Med 91:141–150
[20] Huberty J, Green J, Glissmann C, et al (2019) Efficacy of the Mindfulness Meditation Mobile App ‘Calm’ to Reduce Stress Among College Students: Randomized Controlled Trial. JMIR Mhealth Uhealth 7:e14273
[21] Egilsson E, Bjarnason R, Njardvik U (2021) Usage and Weekly Attrition in a Smartphone-Based Health Behavior Intervention for Adolescents: Pilot Randomized Controlled Trial. JMIR Form Res 5:e21432
[22] Liang Z, Melcer E, Khotchasing K, Hoang NH (2024) Co-design personal sleep health technology for and with university students. Front Digit Health 6:1371808
[23] Peabody JO, Miller DC, Linsell S, et al (2015) 192 Wisdom of the crowds: Use of crowdsourcing to assess surgical skill of robot-assisted radical prostatectomy in a statewide surgical collaborative. European Urology Supplements 2:e192–e192a
[24] Benis A, Tamburis O, Chronaki C, Moen A (2021) One Digital Health: A Unified Framework for Future Health Ecosystems. J Med Internet Res 23:e22189
[25] Petrakaki D, Hilberg E, Waring J (2021) The Cultivation of Digital Health Citizenship. Soc Sci Med 270:113675
[26] Thomason J (2021) Big tech, big data and the new world of digital health. Global Health Journal 5:165–168
[27] Powell J, Deetjen U (2019) Characterizing the Digital Health Citizen: Mixed-Methods Study Deriving a New Typology. J Med Internet Res 21:e11279
[28] Temiz S (2021) Open Innovation via Crowdsourcing: A Digital Only Hackathon Case Study from Sweden. Journal of Open Innovation: Technology, Market, and Complexity 7:39
[29] Oyebode O, Alhasani M, Mulchandani D, et al (2021) SleepFit: A Persuasive Mobile App for Improving Sleep Habits in Young Adults. In: 2021 IEEE 9th International Conference on Serious Games and Applications for Health(SeGAH). ieeexplore.ieee.org, pp 1–8
[30] Tan-MacNeill KM, Smith IM, Weiss SK, et al (2020) An eHealth insomnia intervention for children with neurodevelopmental disorders: Results of a usability study. Res Dev Disabil 98:103573
[31] Cormier M, Orr M, Kaser A, et al (2024) Sleep well, worry less: A co-design study for the development of the SMILE app. Digit Health 10:20552076241283240
[32] Takeuchi H, Suwa K, Kishi A, et al (2022) The Effects of Objective Push-Type Sleep Feedback on Habitual Sleep Behavior and Momentary Symptoms in Daily Life: mHealth Intervention Trial Using a Health Care Internet of Things System. JMIR mHealth and uHealth 10:e39150
[33] Leichman ES, Gould RA, Williamson AA, et al (2020) Effectiveness of an mHealth Intervention for Infant Sleep Disturbances. Behav Ther 51:548–558
[34] Aji M, Gordon C, Stratton E, et al (2021) Framework for the Design Engineering and Clinical Implementation and Evaluation of mHealth Apps for Sleep Disturbance: Systematic Review. J Med Internet Res 23:e24607
[35] Alamoudi D, Nabney I, Crawley E (2024) Evaluating the effectiveness of the SleepTracker app for detecting anxiety- and depression-related sleep disturbances. Sensors (Basel) 24:722
[36] Udd-Granat L, Lahti J, Donnelly M, et al (2023) Internet-delivered cognitive behavioral therapy (iCBT) for common mental disorders and subsequent sickness absence: a systematic review and meta-analysis. Scand J Public Health 51:137–147
[37] Yardley L, Ainsworth B, Arden-Close E, Muller I (2015) The person-based approach to enhancing the acceptability and feasibility of interventions. Pilot Feasibility Stud 1:37
[38] Ahtinen A, Andrejeff E, Vuolle M, Väänänen K (2016) Walk as You Work: User Study and Design Implications for Mobile Walking Meetings. In: Proceedings of the 9th Nordic Conference on Human-Computer Interaction. Association for Computing Machinery, New York, NY, USA, pp 1–10
[39] Brabham DC, Ribisl KM, Kirchner TR, Bernhardt JM (2014) Crowdsourcing applications for public health. Am J Prev Med 46:179–187
[40] Dawson R, Bynghall S (2012) Getting results from crowds
[41] Créquit P, Mansouri G, Benchoufi M, et al (2018) Mapping of Crowdsourcing in Health: Systematic Review. J Med Internet Res 20:e187
[42] Crawford R, Emmerson C (2012) NHS and social care funding: the outlook to 2021/22. http://www.nuffieldtrust.org.uk/files/2017-01/nhs-social-care-funding-outlook-2021-22-web-final.pdf. Accessed 16 Dec 2022
[43] Barony Sanchez RH, Bergeron-Drolet L-A, Sasseville M, Gagnon M-P (2022) Engaging patients and citizens in digital health technology development through the virtual space. Front Med Technol 4:958571
[44] Pedgley O (2007) Capturing and analysing own design activity. Design Studies 28:463–483
[45] Yang Q, Cranshaw J, Amershi S, et al (2019) Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, pp 1–12
[46] (2017) E-ACT Multi-Academy Trust. In: E-ACT. https://www.e-act.org.uk/. Accessed 31 Mar 2024
[47] (2021) Association of Colleges. In: Association of Colleges. https://www.aoc.co.uk/. Accessed 31 Mar 2024
[48] (2020) The Sleep Charity. In: The Sleep Charity. https://thesleepcharity.org.uk/. Accessed 31 Mar 2024
[49] (2023) The McPin Foundation. In: The McPin Foundation. https://mcpin.org/. Accessed 31 Mar 2024
[50] Young People Improving Research. The Bristol Young People’s Advisory Group (YPAG). In: Generation R, Young People Improving Research. https://generationr.org.uk/bristol/. Accessed 20 Mar 2024
[51] Supporting patient and public involvement in research. In: National Institute for Health and Care Research. https://www.nihr.ac.uk/explore-nihr/campaigns/supporting-patient-and-public-involvement-in-research.htm. Accessed 20 Mar 2024
[52] Clarke V, Braun V (2014) Thematic Analysis. In: Encyclopedia of Critical Psychology. Springer, New York, NY, pp 1947–1952
[53] Taylor A, Brigden A, Loades M, Crawley E (2018) Exploring the experience of young people with chronic fatigue syndrome (CFS)/myalgic encephalomyelitis (ME) and depression using qualitative methodology. In: SAGE Research Methods Cases: Part 2
[54] Duffy A, Boroumandzad N, Sherman AL, et al (2025) Examining challenges to co-design digital health interventions with end users: Systematic review. J Med Internet Res 27:e50178
[55] Arvidsson S, Gilljam B-M, Nygren J, et al (2016) Redesign and Validation of Sisom, an Interactive Assessment and Communication Tool for Children With Cancer. JMIR Mhealth Uhealth 4:e76
[56] Rumsfeld JS, Brooks SC, Aufderheide TP, et al (2016) Use of Mobile Devices, Social Media, and Crowdsourcing as Digital Strategies to Improve Emergency Cardiovascular Care: A Scientific Statement From the American Heart Association. Circulation 134:e87–e108
[57] AirOps. In: AirOps. https://www.airops.com/nlp-guide/how-to-classify-instagram-social-media-posts-with-generative-ai. Accessed 23 May 2024
[58] Ridge D, Bullock L, Causer H, et al (2023) ‘Imposter participants’ in online qualitative research, a new and increasing threat to data integrity? Health Expect 26:941–944
[59] Roehl J, Harland D (2022) Imposter participants: Overcoming methodological challenges related to balancing participant privacy with data quality when using online recruitment and data collection. Qual Rep. https://doi.org/10.46743/2160-3715/2022.5475
[60] Park PS, Goldstein S, O’Gara A, et al (2024) AI deception: A survey of examples, risks, and potential solutions. Patterns (N Y) 5:100988
[61] Guo C, Ashrafian H, Ghafur S, et al (2020) Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. npj Digital Medicine 3:1–14
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Anthony Duffy, Sarah E Bennett, Lucy Yardley, Sylvain Moreno
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited.
