Crowdsourced Perceptual Ratings of Voice Quality in People With Parkinson’s Disease Before and After Intensive Voice and Articulation Therapies: Secondary Outcome of a Randomized Controlled Trial
Tara McAllister, Christopher Nightingale, Gemma Moya-Galé, Ava Kawamura, Lorraine Olson Ramig
What is it about?
This study investigated whether untrained listeners who are recruited through online crowdsourcing platforms can assess voice treatment for people with Parkinson’s Disease (PD) in a way that agrees with reliable measurement tools. In crowdsourcing, researchers use digital platforms to connect with a large number of individuals who perform tasks online for monetary compensation.
This study aimed to replicate a 2022 study (Moya-Galé et al., Journal of Voice) that showed improved voice quality for people with PD after receiving the Lee Silverman Voice Treatment (LSVT LOUD). The results of the original study were determined using the Acoustic Voice Quality Index, a metric that combines multiple acoustic measures of speech to arrive at a single value that correlates highly with expert listeners’ ratings.
We wanted to know if those acoustically measurable changes would also be apparent to listeners with no special training related to speech or PD. Thirty adults recruited through Amazon’s Mechanical Turk crowdsourcing platform listened to voice recordings from the Moya-Galé et al. study and were asked to classify each speech sample as typical or atypical with respect to voice quality. We found that the crowdsourced listeners’ ratings replicated the original study’s finding: significant gains in voice quality were observed after LSVT LOUD treatment but not after two comparison treatments. We also found a significant correlation between crowdsourced listeners’ ratings and AVQI.
Why is it important?
Studies of speech and voice treatment methods for individuals with PD typically evaluate progress using fine-grained acoustic measures or highly trained listeners’ perceptual ratings. However, this approach leaves some uncertainty about the real-world impact of these changes, since it is not always clear they would be apparent to untrained listeners that patients interact with in their daily life. Online crowdsourcing gives us an opportunity to assess the outcomes of a treatment study in a way that is more directly relevant to patients’ lived experience.
Perspectives
For additional perspectives on this treatment study and line of treatment research, we asked lead author of this study Tara McAllister, Ph.D., CCC-SLP a few questions. Her insights are provided below.
Why did you want to explore this topic?
I have used crowdsourcing in the past as part of my work on children with speech sound disorder – we ask online listeners to rate children’s speech productions as correct or misarticulated. However, there was a lack of research applying crowdsourcing in other domains. We were particularly interested in studying voice quality (whether a person’s voice sounds hoarse, breathy, or typical) because it isn’t a concept that comes up often in everyday life, and we wanted to know if crowdsourcing would still produce valid results in this unfamiliar context.
What were the key take away points from this study?
These results support the use of crowdsourcing to evaluate speech samples collected from clinical populations, even for constructs such as voice quality that may be less familiar to untrained listeners. They also add to the body of evidence showing that LSVT LOUD treatment is an effective means to improve speech and voice in patients with PD.
How might this impact SLPs who are working people with Parkinson Disease?
We think that crowdsourcing has exciting potential to make it easier for researchers to understand the real-world relevance of the treatment methods they study, and ultimately support better alignment of priorities between researchers, clinicians, and patients.
Were there any surprises or unexpected outcomes you learned?
When we were doing pilot testing to prepare for the study, we were surprised how challenging it can be to rate speakers’ voice quality! Everyone’s voice is different in terms of pitch and intonation, and voice quality changes with age even in healthy adults, so it can be very tricky to decide where to draw the line between typical and atypical. That made it all the more impressive to see that untrained listeners’ ratings had good agreement with a precisely calibrated acoustic measure like the AVQI. It really speaks to the idea of “the wisdom of crowds” – any one person in the crowd may not be able to give you the exact answer, but when you average over many people, the result can be impressively accurate!
What happens next in terms of your research on LSVT LOUD and Parkinson’s?
There is exciting work ongoing in the use of artificial intelligence (AI) to evaluate speech in patients with PD and other communication disorders. Historically, crowdsourcing has played an important role in AI – specifically, crowdsourced participants are recruited to rate or classify large sets of data that are then used to train AI models. I look forward to bringing our experience with online crowdsourcing to bear on the challenges presented by this new technological frontier.
About the Author
Tara McAllister, Ph.D., SLP-CCC
Tara McAllister is an Associate Professor of Communicative Sciences and Disorders at New York University. Her research aims to understand how speech skills are acquired in both typical and clinical populations, and why developmental speech patterns resolve in some individuals but persist in others. Dr. McAllister leads NIH-supported research investigating acoustic and ultrasound biofeedback intervention for residual speech errors, and she directs the development of the staRt app for visual-acoustic biofeedback.