A new study from NYU Langone Health finds that generative artificial intelligence compares favorably with healthcare providers’ responses when answering patient questions in electronic health record inboxes. Plus, it could reduce the documentation burden for clinicians.
The study, conducted by researchers at New York University’s Grossman School of Medicine, found that a genAI messaging tool not only generated accurate answers to patient questions in the electronic health record, but the responses also showed greater empathy.
“Our results suggest that chatbots can reduce healthcare providers’ workload by enabling efficient and empathetic responses to patients’ concerns,” Dr. William Small, clinical assistant professor in the NYU Grossman Department of Medicine and lead author of the study, said in an announcement Tuesday.
WHY IT MATTERS
According to NYU Langone Chief Medical Information Officer Dr. Paul Testa, it was common for physicians to receive more than 150 In Basket messages per day during the pandemic. This represents a year-over-year increase in daily messages of more than 30%.
NYU Langone has been looking at how to address notebloat and other documentation burdens that are major sources of physician burnout. For this new study, medical researchers asked 16 primary care physicians to rate 344 randomly assigned pairs of AI and human responses to patient messages for accuracy, relevance, completeness, and tone.
According to the researchers in the report Large Language Model–Based Responses to Patients’ In-Basket Messages, published in JAMA, the generative AI responses outperformed human caregivers by 9.5% in terms of comprehension and tone.
For the blind studythe participating physicians indicated whether they would use the AI response as a first draft or start over. They did not know which responses humans or the AI tool had compiled.
According to the researchers, the AI responses were 125% more likely to be perceived as empathetic and 62% more likely to use language that conveyed positivity and connectedness.
However, they were 38% longer and 31% contained more complex language.
“While humans responded to patient questions at a 6th grade level, AI wrote at an 8th grade level, according to a standard measure of readability called the Flesch Kincaid score,” the researchers said.
They also noted that more research is needed to confirm whether specific private data improved the performance of AI tools.
THE BIGGER TREND
In 2023, NYU Langone licensed GPT4, allowing physicians to experiment with real patient data in a secure environment.
Also last year, the healthcare research organization encouraged teams of clinicians, educators and researchers to collaborate to test large language models in a Generative AI Prompt-A-Thon in Healthcare, which required no experience.
During the event, which attracted 70 attendees and more than 500 others via a live webinar, the health system wanted to use the observations generated to inform internal capacity building in generative AI.
Researchers later used AI to review transcripts of 820 individuals who received psychotherapy during the first wave of COVID-19 in the United States and compare them to those of healthcare workers.
The AI tool could detect stress in overworked hospital workers: people who discussed sleep deprivation or mood problems during therapy sessions were more likely to be diagnosed with anxiety and depression.
ON THE RECORD
“We found that AI chatbots integrated into the EHR and using patient-specific data can craft messages that are comparable in quality to those of human caregivers,” Small said in a statement.
“Now that physician approval has been granted, GenAI messages will soon be on par with human-generated responses in terms of quality, communication style, and usability,” said corresponding author Dr. Devin Mann, senior director of informatics innovation at NYU Langone Medical Center Information Technology.
Andrea Fox is Editor-in-Chief of Healthcare IT News.
Email address: afox@himss.org
Healthcare IT News is a publication of HIMSS Media.