NYU Langone Health’s MCIT Department of Health Informatics, Institute for Innovation in Medical Education and Institute for Excellence in Health Equity held their first Genative AI Prompt-A-Thon in Health Care this past month.
During the event, teams of physicians, educators and researchers worked together to find artificial intelligence-based solutions to healthcare challenges using real, anonymized patient data.
The event focused on large language models (LLMs) that predict likely options for the next word in any sentence, paragraph or essay, based on how real people used words in context billions of times in documents on the Internet.
LLM systems, also called generative AI, randomly fill in a mix of likely next words to give a sense of variety and creativity. A side effect of this next-word prediction is that the models are ‘skilled’ at summarizing long texts, extracting important information from databases and generating human-like conversations like chatbots.
Despite these advances, such AI programs don’t think and can produce conclusions and references that don’t exist, Prompt-A-Thon organizers said. Therefore, they require close supervision by human users, especially in healthcare, where the technology has the potential to increase safety and improve care.
THE PROBLEM
“The problem we faced was how to engage members of our workforce who may have transformative applications for generative AI and may not be technologically savvy enough to participate in our other capacity building initiatives, such as exploratory access or guided projects,” said Dr. Jonathan Austrian. , associate chief medical information officer, intramural informatics, at NYU Langone Health.
“These other initiatives worked well for highly motivated colleagues who only needed our HIPAA-compliant, patient-safe NYU GPT to safely experiment with real-world clinical data or proprietary research ideas,” he explained. “Our guided projects were ideal for researchers, educators and clinicians who already had a better idea to leverage generative AI and needed mentorship from our data scientist team to take their ideas to the next level.”
The gap was frontline clinicians, researchers, educators, and operational leaders who understand the issues facing the healthcare system, but need focused time and personalized support to connect generative AI to these challenges. There was significant demand from the workforce to close this gap, and the fastest and most efficient way to meet that demand was through an event called a prompt-a-thon, Austrian said.
PROPOSAL
The Genative AI Prompt-A-Thon in Health Care was a mechanism to quickly engage a large portion of NYU Langone Health’s workforce in generative AI and, in parallel, expose the health care system’s existing program of engagement initiatives to those not could be accommodated by the prompt-a-thon.
The prompt-a-thon aimed to lower barriers for the workforce to engage in generative AI. Staff emphasized the personal nature of the event, mentorship from generative AI experts, and that no prior experience with generative AI was required.
“Sometimes I felt like Captain Kirk was speaking to our engineering team, ‘Scotty, we need more computing power!’
Dr. Jonathan Austrian, NYU Langone Health
“In addition to involving more of our workforce in generative AI, we also felt that such an in-person event, combining different specialties, roles and experiences, could create new ideas and relationships that drive innovation: a true learning community,” said Austrian.
RESULTS
The health care system will use the results of the Generative AI Prompt-A-Thon in many ways.
“First, the 70 people who attended in person and the more than 500 people who watched the webinar remotely will be included in our community of learners who we will continue to engage with access to GPT, updates on available technologies, and additional approaches to generative to leverage AI,” Austrian explained.
“Second, we expect that many of the ideas generated during Prompt-A-Thon will be further refined by our community and ultimately evolve into applications operationalized at NYU and distributed around the world,” he continued.
Third, direct observations by mentors and the results of the research conducted will inform how the healthcare system continues to build its internal capacity to leverage generative AI.
“We have invited our health sciences library staff to observe the workshops because they will be working with us to formalize a curriculum in generative AI,” says Austrian. said. “Based on the success of the event, we will do Prompt-A-Thons along the way with smaller groups of researchers, educators, clinicians and members of our corporate services.
“And finally, we learned a lot about the technology infrastructure needed to support scalable, intensive use of generative AI,” he said. “Specifically, we had 70 people querying NYU GPT synchronously. At the same time, we had our data scientist team in our command center observe these interactions in real time to understand any error messages or processing delays.”
NYU Langone Health partners at Microsoft were also on site to ensure participants had a seamless technology experience and that the healthcare system can scale the experience as usage increases.
The preliminary results of the study speak to the prompt-a-thon’s impact on attendees. Of the 62 who responded, 90% believe the prompt-a-thon increased how efficiently they could do their work with generative AI. Eighty-four percent said they would likely submit a healthcare-related generative AI project.
ADVICE FOR OTHERS
Austrian certainly recommends that other healthcare organizations consider an event similar to this one Generative AI Prompt-A-Thon in Healthcare.
“For the introductory discussions, we spent a lot of time describing the capabilities of generative AI and the important ethical and trust issues to consider when using generative AI,” he recalls. “Next time we’ll spend a little more time on the basics of rapid engineering.
“We had to find a balance between involving our mentors in the enabling groups without stifling innovation or disrupting group dynamics,” he continued. “We opted for one mentor for every two teams of four people per team. Given how new generative AI is, I recommend staffing one mentor per team. The ‘blank’ prompt page was overwhelming at first for some of our groups .”
Finally, Austrian said he can’t overestimate the importance of a strong technology infrastructure to enable a prompt-a-thon.
“Our IT department spent a lot of time stress-testing NYU GPT and developing creative solutions to distribute the load across all users,” he concluded. “Sometimes I felt like Captain Kirk was speaking to our engineering team, ‘Scotty, we need more computing power!'”
Follow Bill’s HIT coverage on LinkedIn: Bill Siwicki
Email him: bsiwicki@himss.org
Healthcare IT News is a HIMSS Media publication.