Hirsch T, Soma C, Merced K, et al. “It’s hard to argue with a computer:” Investigating psychotherapists’ attitudes towards automated evaluation. Proceedings of the 2018 Designing Interactive Systems Conference; 2018; Hong Kong, China. doi: 10.1145/3196709.3196776e
Researchers recruited novice (n=10) and experienced (n=11) counselors to deliver a mock session of motivational interviewing (MI) while being recorded by the Counselor Observer Ratings Expert for Motivational Interviewing (CORE-MI). CORE-MI uses machine learning to code counselor speech for MI adherent (e.g. reflections, open questions) and non-adherent (e.g., unsolicited advice, confrontational statements) behavior. CORE-MI calculates scores for MI adherence (ratio of adherent to non-adherent behaviors), spirit (global rating of clinician supportiveness), empathy (global rating of clinician efforts to understand client perspectives), and fidelity (composite rating of calculated metrics and behavior counts). CORE-MI provides counselors with summaries visualizing the evaluated metrics, behavior counts, and session recordings, color coded by speaker and speaker intensity. Participants were interviewed about their perceptions of CORE-MI performance summaries and use in practice. Ten participants reported the layout of the summary was clean and simple, but some reported difficulty interpreting the meaning of the measures. Novice counselors seemed willing to accept the evaluation due to its perceived objectivity, but experienced counselors seemed more likely to question evaluations that did not match perceptions of their performance. Participants reported wanting to use the information to track their performance over time. Summaries evoked participant reflections on their use of MI techniques (18), perceptions of their performance (9), and areas for improvement (7). All participants felt CORE-MI could be used for feedback and supervision, particularly to guide conversations between trainees and supervisors, but 4 counselors worried that the tool could be misused in workplaces.