Current models emphasize attention to implementation throughout the intervention development process. Our goal in compiling these instruments, is to promote harmonized measurement of implementation at all stages of digital therapeutic intervention research. Instrument selection was guided by implementation science frameworks (Damschroder et al., 2009; Procter et al 2011; Rogers, 2003), existing repositories of evidence-based instruments (Lewis et al., 2015; National Cancer Institute, 2018), and demonstrated strong psychometric properties in published empirical literature. For more information about integrating implementation measurement into a stage model of digital intervention development (Onken et al., 2013), see below.
Implementation Outcomes
Acceptability
Unified Theory of Acceptance and Use of Technology (UTAUT)
Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Toward a unified view. MIS Quarterly. 2003;27(3):425-478.
A 31-item measure of intervention acceptability among healthcare professionals. The UTAUT is an update to the Technology Acceptance Model (TAM; Davis, 1989). Grounded in the Theory of Planned Behavior, subscales include: performance expectancy (usefulness), effort expectancy (ease of use), attitudes toward using the technology, feelings of self-efficacy and anxiety for using technology, social influence and facilitating conditions for using the technology, and behavioral intentions to use the technology in the future. Items are measured using a 7-point Likert scale (Strongly Disagree-Strongly Agree). Scoring is calculated mean of subscales or total.
Unified Theory of Acceptance and Use of Technology 2 (UTAUT2)
Venkatesh V, L. Thong JY, Xu X. Consumer acceptance and use of information technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly. 2012;36(1):157-178
An update of the UTAUT, this is a 28-item measure of patient and consumer perceptions of intervention acceptability.
Acceptability of Intervention Measure (AIM)
Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implementation Science. 2017;12(1):108. doi: 10.1186/s13012-017-0635-3
A 4-item measure of perceived intervention acceptability. Items are measured on a 5-point Likert scale (Completely Disagree-Completely Agree). Score is calculated mean.
Appropriateness
Intervention Appropriateness Measure (IAM)
Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implementation Science. 2017;12(1):108. doi: 10.1186/s13012-017-0635-3
A 4-item measure of perceived intervention appropriateness. Items are measured on a 5-point Likert scale (Completely Disagree-Completely Agree). Score is calculated mean.
Cost
Drug Abuse Treatment Cost Analysis Program (DATCAP): Client (Outpatient/Inpatient)
French, M.T. (2005). Drug Abuse Treatment Cost Analysis Program (DATCAP): Client (Outpatient/Inpatient) Version Third Edition, University of Miami, Coral Gables, Florida.
A 17-item (inpatient) or 20-item (outpatient) measure of costs incurred by patients receiving inpatient or outpatient substance use treatment. Measure targets identification of costs incurred by the patient related to treatment program, transportation, time away from work, and other costs. A program version is also available, see the DATCAP website for more information.
Non-Study Medical Services
Polsky D, Glick HA, Yang J, Subramaniam GA, Poole SA, Woody GE. Cost-effectiveness of Extended Buprenorphine-Naloxone Treatment for Opioid-Dependent Youth: Data from a Randomized Trial. Addiction (Abingdon, England). 2010;105(9):1616-1624. doi:10.1111/j.1360-0443.2010.03001.x
A 6-item instrument to assess medical costs incurred by study participants outside of study-related services. Measure assesses utilization of mental health care, physical health care, detoxification programs, hospital admissions, and emergency room services. Interpretation of this instrument requires developing a list of costs associated with services assessed.
European Quality of Life Measure – 5 Dimensions – 3 levels (EuroQOL-5D-3L)
Brooks R. EuroQol: The current state of play. Health Policy. 1996;37(1):53-72. doi: 10.1016/0168-8510(96)00822-6
A 6-item quality of life measure for patients and consumers. The EuroQOL can be administered in conjunction with a measure of cost (e.g. the DATCAP) to facilitate calculation of quality adjusted life years. Items measure quality of life regarding mobility, self-care, usual activities, pain/discomfort, and anxiety/depression, as well as overall health. Items are measured on a 3-point Likert scale (No Disability-Extreme Disability). Scoring is mean scores for individuals or groups. More information about scoring the EuroQOL-5D-3L is available in the user manual on the EuroQOL website.
Feasibility
Feasibility of Intervention Measure (FIM)
Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implementation Science. 2017;12(1):108. doi: 10.1186/s13012-017-0635-3
A 4-item instrument to assess perceived intervention feasibility. Items are measured on a 5-point Likert scale (Completely Disagree-Completely Agree). Score is calculated mean.
Sustainability
Program Sustainability Assessment Tool
Luke DA, Calhoun A, Robichaux CB, Elliot MB, Moreland-Russell S. The Program Sustainability Assessment Tool: A new instrument for public health programs. Preventing Chronic Disease. 2014;11. doi: 10.5888/pcd11.130184
A 40-item measure of intervention sustainability as reported by organization provider and administrative stakeholders. Subscales include: political support, funding stability, partnerships with stakeholders, organizational capacity, program evaluation, program adaptation, communication with stakeholders, and strategic planning. Items are measured on a 7-point Likert scale (Little or No Extent-Very Great Extent). The measure website hosts online and PDF versions of the Program Sustainability assessment and instructions for scoring. The measure is scored by calculating subscale means and totals.
Implementation Contexts: Characteristics of Intervention
Usability
System Usability Scale (SUS)
Brooke J. SUS-A quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, eds. Usability Evaluation in Industry. London: Taylor & Francis; 1996:189–194.
A 10-item measure of digital intervention usability. Items are scored on a 5-item Likert scale (Strongly Disagree-Strongly Agree). Scoring the SUS requires completing simple calculations to convert the final score to be on a scale from 0-100. More information about the SUS can be found on usability.gov.
Perceived Intervention Characteristics
Perceptions of Computerized Therapy Questionnaire-Patient (PCTQ-P)
Carper MM, McHugh RK, Murray HW, Barlow DH. Psychometric Analysis of the Perceptions of Computerized Therapy Questionnaire-Patient Version (PCTQ-P). Administration and Policy in Mental Health and Mental Health Services Research. 2014;41(1):104-113. doi: 10.1007/s10488-012-0440-x
A 30-item instrument to assess patient/consumer perceptions of intervention characteristics. Subscales include relative advantage, compatibility, ease of use, observability, trialability, and future use intentions. Items are measured on a 7-point Likert scale (Strongly Disagree-Strongly Agree). The measure is scored by calculating subscale means.
Perceived Characteristics of Innovating Questionnaire (PCI)-Short Version
Moore GC, Benbasat I. Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation. Information Systems Research. 1991;2(3):192-222. doi: 10.1287/isre.2.3.192
A 25-item measure to assess organization provider and administrative stakeholder perceptions of intervention characteristics. Subscales include: voluntariness, relative advantage, compatibility, image (impact on social standing), ease of use, result demonstrability, observability, and trialability. Though most PCI items assess characteristics of interventions in the work place, the result demonstrability subscale is general enough that we suggest using this scale with patients and consumers, as this construct is not included in the PCTQ-P. Items can be measured on a 7-point Likert scale (Strongly Disagree-Strongly Agree). The measure can be scored by calculating subscale means or totals.
Compatibility Beliefs in Technology Questionnaire
Karahanna E, Agarwal R, Angst CM. Reconceptualizing Compatibility Beliefs in Technology Acceptance Research. MIS Quarterly. 2006;30(4):781-804. doi: 10.2307/25148754
A 21-item measure of perceptions of compatibility of interventions for health care professionals. Subscales include: compatibility with existing practices, compatibility with prior experience, compatibility with values, ease of use, usefulness, usage intensity, and usage scope. Items are measured on a 7-point Likert scale (Strongly Disagree-Strongly Agree). The measure is scored by calculating subscale means.
User Engagement Scale (UES) -Long Form
O’Brien HL, Cairns P, Hall M. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies. 2018;112:28-39. doi: 10.1016/j.ijhcs.2018.01.004
A 30-item measure of user engagement with interventions. Subscales include: focused attention, perceived usability, aesthetics, and reward. Items are measured on a 5-point Likert scale (Strongly Disagree-Strongly Agree). The measure is scored by calculating subscale means. A short-form version was iteratively developed in 2018, but has not yet been thoroughly evaluated.
Implementation Contexts: Characteristics of Inner Setting
Implementation Leadership
Implementation Leadership Scale
Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implementation Science. 2014;9(1):45. doi: 10.1186/1748-5908-9-45
A 12-item measure of implementation leadership among health care professionals with versions for organization staff and organization leadership. Subscales include: proactive (supervisor initiative), knowledgeable (supervisor knowledge), supportive (support from supervisor), and perseverant (supervisor perseverance). Items are measured on a 5-point Likert scale (Not At All-Very Great Extent). Score by calculating subscale means or an average of scale means for total score.
Implementation Climate
Implementation Climate Scale
Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implementation Science. 2014;9(1):157. doi: 10.1186/s13012-014-0157-1
An 18-item measure of perceptions of implementation climate among organization stakeholders. Subscales include: focus on evidence-based practice, educational support for evidence-based practice, recognition for evidence-based practice, rewards for evidence-based practice, selection for evidence-based practice, selection for openness. Items are measured on a 5-point Likert scale (Not At All-Very Great Extent). Score by calculating subscale means or an average of scale means for total score.
Organizational Readiness for Implementation
Organizational Readiness for Implementing Change
Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: A psychometric assessment of a new measure. Implementation Science. 2014;9(1):7. doi: 10.1186/1748-5908-9-7
A 12-item instrument to assess organization stakeholder perceptions of organizational readiness for implementation. Items are measured on a 5-point Likert scale (Disagree-Agree). Score is calculated mean.
References
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science 2009;4(50). doi: 10.1186/1748-5908-4-50
- Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implementation Science 2015;10(1):155. doi: 10.1186/s13012-015-0342-x
- National Cancer Institute. GEM-Dissemination and Implementation Initiative (GEM-D&I). https://www.gem-measures.org/public/wsmeasures.aspx?cat=8&aid=1&wid=11, 2018.
- Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning Clinical Science. Clinical Psychological Science 2013;2(1):22-34. doi: 10.1177/2167702613497932
- Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research 2011;38:65-76. doi: 10.1007/s10488-010-0319-7
- Rogers EM. Diffusion of Innovations. 5th ed. New York, NY: Free Press; 2003. 0743222091