Machine learning can be used to comb through online reviews of substance use treatment facilities to home in on qualities that are important to patients but remain hard to capture via formal means, such as surveys, researchers from the Perelman School of Medicine at the University of Pennsylvania show. The researchers found that professionalism and staff dedication to patients were two of the top qualities that could be attributed to either a negative or positive review of the facility. Findings from this study were published today in the Journal of General Internal Medicine.
“Searching for—and connecting with—therapy can be very difficult and confusing. Many individuals start their search online, where they are likely to see an online review accompanying other information about a treatment facility,” said the study’s lead author, Anish Agarwal, MD, a clinical innovation manager in the Penn Medicine Center for Digital Health and an assistant professor of Emergency Medicine. “These online reviews can provide commentary on what is driving positive, or negative, patient experiences throughout recovery, but they must be accurately identified. Through machine learning, we’ve shown that this is possible, and we hope such findings can be used to improve patient-centered addiction care.”
Currently, there are no nationwide measures of quality to evaluate and compare facilities that treat substance use disorder. In the past, Agarwal—along with this study’s co-author, Sharath Guntuku, Ph.D., a researcher in the Center for Digital Health and an assistant professor of Computer Science, and senior author Raina Merchant, MD, the director of the Center for Digital Health—analyzed reviews on Google and Yelp to see whether a national survey offered by the Substance Abuse and Mental Health Services Administration (SAMHSA) to inventory services being offered was also able to gauge patient satisfaction (it largely didn’t). So the team set out to use a similar technique to gauge what drives positive or negative experiences with facilities through the unfiltered lens of Yelp.
“We felt that this would provide a great deal of insight into the patient experience,” Guntuku said. “Tapping into user-generated reviews offers a way to understand their narrative.”
To accomplish this, the researchers pulled reviews from SAMSHA-recognized facilities that had been reviewed at least five times. This amounted to more than 500 facilities across the United States. Text of the reviews were then run through a natural language processing algorithm powered by machine learning. Through this, topics were identified and collected from within the reviews. The researchers then were able to categorize them thematically.
Overall, the researchers classified 16 recurring themes in reviews. When it came to the positive reviews, the top themes were “long-term recovery,” “dedicated staff,” and “dedication to patients.” The top three themes found in negative reviews were “professionalism,” “phone communication,” and “overall communication.”
The fact that the top themes for both positive and negative reviews had to do with the conduct and commitment of facility staff does not come as a great surprise.
“Dedication and professionalism are critical to the recovery experience,” Agarwal said. “Having trusted and approachable staff who care about individuals is the crux of all health care, but it is likely underscored in addiction.”
Other top themes on the positive side were “group therapy experience” and “inpatient rehabilitation,” while “wait times in facility” and “management” rounded out the top five most prevalent themes among negative reviews.
“We are still relatively early in this research, but we’re getting a direct look into some core values that substance use treatment facilities could use to guide and improve their treatment,” Merchant said. “This feedback is tremendously valuable, and we’re showing that it can be distilled effectively into key themes through the use of machine learning.”
Agarwal said he and the other researchers are hoping to explore more ways that patients, families, and their support networks think about substance use treatment and health care as a whole.
Source: Read Full Article