Hermes EDA, Lyon AR, Schueller SM, Glass J. (2019). Measuring the implementation of behavioral intervention technologies: Recharacterization of established outcomes. Journal of Medical Internet Research. 21(1): e11752. doi: 10.2196/11752
Researchers examined outcomes of implementation research (i.e., acceptability, adoption, appropriateness, feasibility, fidelity, cost, reach, sustainability) applied to digital behavioral health interventions. Target respondents for measurement of implementation outcomes will vary based on which stakeholders are involved in digital intervention delivery. Researchers highlighted acceptability and usability as overlapping, but distinct concepts. While acceptability is an implementation outcome, researchers describe usability as an outcome of intervention development. Appropriateness also has some overlap with usability. Appropriateness may also capture different concepts in early (e.g. usability) compared to later (e.g. organization technology infrastructure, interoperability) implementation. Adoption and fidelity can be measured using metrics tracked by the intervention technology, so it is important for researchers to define each of these constructs before implementation. Feasibility can also be measured using data collected by the digital intervention, which could elucidate awareness of intervention features, workflows within the intervention, and workarounds for difficulties as they are encountered. Researchers highlight the overlap between costs of development and implementation of digital behavioral health intervention since maintenance of technology can involve ongoing updates. Likewise, sustainability, especially in the case of fully automated interventions, may extend beyond continued use by the consumers to include continued budgeting for advertising and maintaining the intervention by the developer.