Texas AG settles lawsuit with clinical genAI company
Texas Attorney General Ken Paxton has announced a settlement with Dallas-based artificial intelligence developer Pieces Technologies, resolving allegations that the company’s generative AI tools jeopardized patient safety by overpromising on accuracy.
WHY IT IS IMPORTANT
The Irving, Texas-based company uses generative AI to summarize real-time electronic medical record data about patient conditions and treatments. The software is used in at least four hospitals in the state, according to the settlement.
The company advertised a “severe hallucination rate” of less than one in 100,000, the settlement agreement.
While Pieces denies any wrongdoing or liability and argues it did not violate the Texas Deceptive Trade Practices-Consumer Protection Act, the settlement with the AG requires the company to “clearly and conspicuously” disclose the meaning or definition of that metric and describe how it is calculated — or else “hire an independent, third-party auditor to evaluate, measure, or substantiate the performance or characteristics of its products and services.”
Pieces agreed to abide by the settlement terms for five years, but told Law360 on Wednesday that the Texas attorney general had approved it.
Healthcare IT News has contacted the company for comment and will update this story if it responds.
THE BIGGER TREND
As artificial intelligence (AI) – particularly genAI – becomes more widely adopted in hospitals and healthcare systems, the challenges surrounding model accuracy and transparency are becoming more significant. This is especially true as these models make their way into clinical settings.
A recent study from the University of Massachusetts Amherst and Mendel, an AI company focused on AI hallucination detection, found that different types of hallucinations appear in AI-summarized medical records, according to an August report. report in Clinical Trials Arena.
Researchers asked two large language models – Open AI’s GPT-4o and Meta’s Llama-3 – to generate medical summaries from 50 detailed medical notes. They found that GPT had 21 summaries with incorrect information and 50 with generalized information, while Llama had 19 errors and 47 generalizations.
As more AI tools are developed that generate summaries from electronic health records and other medical data, their reliability remains questionable.
“I think we’re at a point where generative AI is not yet transparent, consistent, and reliable,” Dr. John Halamka, chair of the Mayo Clinic Platform, told Healthcare IT News last year. “So we have to be a little bit careful about the use cases we choose.”
To better assess AI, the Mayo Clinic Platform has developed a risk classification system to qualify algorithms before they are used externally.
Dr. Sonya Makhni, medical director of the platform and senior associate consultant for the Department of Internal Medicine at Mayo Clinic Hospital, explained that when thinking about the safe use of AI, healthcare organizations “need to consider how an AI solution could impact clinical outcomes and what the potential risks are if an algorithm is inaccurate or biased or if actions taken on an algorithm are inaccurate or biased.”
She said it is “the responsibility of both the solution developers and the end users to best assess an AI solution in terms of risk.”
ON THE RECORD
“AI companies that offer products used in high-risk environments owe it to the public and their customers to be transparent about their risks, limitations, and appropriate uses,” Texas Attorney General Ken Paxton said in a statement about the settlement with Pieces Technologies.
“Hospitals and other healthcare institutions should consider whether AI products are appropriate and train their employees accordingly,” he added.
Andrea Fox is Editor-in-Chief of Healthcare IT News.
Email address: afox@himss.org
Healthcare IT News is a publication of HIMSS Media.