Baumel A, Edan S, Kane J. (2019). Is there a trial bias impacting user engagement with unguided e-mental health interventions? A systematic comparison of published reports and real-world usage of the same programs. Translational Behavioral Medicine. 9(6): 1020–1033. doi: 10.1093/tbm/ibz147
Researchers conducted a systematic comparison of trial-based and real-world use of unguided digital mental health interventions. To identify popular mobile apps (n = 293) and web-based interventions (n = 62), researchers entered 20 mental health-related search terms (e.g. depression, bipolar, PTSD) into Google and Google Play. Researchers searched medical databases for related research papers using the name of the app or web-based intervention and 10 search terms including usage, adherence, and user engagement. Out of 249 identified research papers, 13 papers met inclusion criteria (e.g. included quantified engagement data, did not offer users direct compensation for program use, and did not use study personnel to remind users to engage with the program). The 13 research papers represented 13 different studies covering 7 apps and 3 web-based interventions. Researchers obtained real-world usage data for the 7 apps and 3 web-based interventions from a web analytics company. The 13 studies and real-world data contained 14 comparable usage metrics (including number of daily sessions, daily minutes of use, monthly visits, and visit duration). Ten of the 13 studies (77%) included proactive recruitment (compared with 3 studies in which the intervention platform automatically recorded native users), human contact with study personnel, and participant assessment procedures. The median intervention usage rate was over 4 times higher in recruited studies than in real-world usage of the same intervention. Study design, severity of symptoms, and in-person versus remote assessment did not have a significant effect on usage. Results suggest that trial-related procedures such as proactive recruitment, human contact, and participant assessment significantly increase user engagement with app and web-based mental health interventions, compared with real-world usage of the same interventions.