CrowdStrike all but assures Capitol Hill: never again
Adam Meyers, senior vice president of Counter Adversary Operations at CrowdStrike, testified before the U.S. House of Representatives on Tuesday, apologizing for the global technical outage last summer that crippled millions of Microsoft Windows computers in healthcare and other industries.
During his appearance before the House Committee on Homeland Security’s Cybersecurity and Infrastructure Protection Subcommittee, Meyers answered technical questions from lawmakers and provided assurances about updated protocols.
While some members of the subcommittee acknowledged the company’s “humility” in addressing the mistake after the incident, they asked pointed questions about what the company has done since then to ensure the incident does not happen again.
WHY IT IS IMPORTANT
Rep. Eric Swalwell (D-Calif.), co-chair of the subcommittee, emphasized the need for a high level of assurance that CrowdStrike will have rigorous quality assurance processes in place.
“A global IT outage that affects every sector of the economy is a disaster you would expect to see in a movie,” said Mark Green, chairman of the House Committee on Homeland Security.
“To make matters worse, the largest IT outage in history was the result of an error,” he said. “As a result, many Americans and allies around the world were unable to call 911, were grounded while traveling, or had their health care services delayed as elective medical procedures were canceled.”
“We have conducted a full review of our systems and have begun implementing plans to improve our content update procedures so that we emerge from this experience as a stronger company,” Meyers assured in his testimony.
CrowdStrike Falcon uses artificial intelligence and machine learning models to anticipate the latest advanced cyberthreats and then pushes content updates to its customers so their systems can recognize and defend against these threats.
In Crowdstrike’s External Technical Root Cause AnalysisOn August 6, the company unveiled its findings and measures to ensure the flaw — which Green said disabled 8.5 million devices and caused $5.4 billion in losses — does not happen again.
One key point Meyers raised during the 90-minute session is the now phased rollout of such updates to minimize widespread disruptions, such as system crashes.
Content updates “that pass the canary test are gradually promoted to broader deployment rings or rolled back if issues are detected,” Crowdstrike said in the analysis, which Meyers described to lawmakers as concentric rings.
Green acknowledged that Crowdstrike has “taken the right stance” since the incident, but he wanted to know if artificial intelligence initiated the update. It didn’t, Meyers explained.
Congressman Mike Ezell, Republican of Mississippi, questioned why a manual solution was needed to get systems up and running again, given the skilled labor shortage in many areas, such as his rural district.
Meyers said that while he drove 10 hours to get a customer up and running again, the bulk of the recovery occurred when Crowdstrike implemented an automated process the next day.
Morgan Luttrell, Republican of Texas, wanted to know, “Where exactly did (the Content Validator) fail?”
“The validator itself has been in use for over a decade and we release 10-12 of these updates every day,” Meyers explains.
“It tested clean, or good, and that’s why it was allowed to be rolled out,” Meyers said, explaining what was explained as a logic error in the company’s Aug. 6 analysis. But, he said, a line with faulty logic that caused the sensor to fail would now be detected under Crowdstrike’s revised procedures.
THE BIGGER TREND
The faulty update pushed to Windows early Friday morning in July caused millions of computers to crash and display the infamous “blue screen of death,” disrupting healthcare delivery in hospitals, health systems and medical practices across the U.S. and around the world.
Providers began working manually to provide patient care without access to electronic medical records and other mission-critical IT systems. While most affected healthcare institutions recovered from the CrowdStrike outage within days, the incident highlighted how third-party technology disruptions can impact patient care.
“This incident demonstrates the interconnectedness of our broad ecosystem: global cloud providers, software platforms, security vendors, and other software vendors and customers,” Microsoft acknowledged in a July 20 statement.
In a July contribution for Healthcare IT NewsChristopher Frenz, information security officer and AVP of IT Security at Mount Sinai South Nassau, said the CrowdStrike outage should be a wake-up call for healthcare system IT and security leaders: Security controls don’t just fail during major events; they’re always at risk. That’s why hospitals and other providers need to invest in security architectures that provide resilience, he said.
ON THE RECORD
“On behalf of everyone at CrowdStrike, I want to apologize,” Meyers said in his statement to lawmakers. “We deeply regret that this happened and are committed to preventing it from happening again. We appreciate the incredible efforts of our customers and partners who, along with our teams, immediately mobilized to restore systems and get many back online within hours.”
“I can assure you that we continue to address this with a great sense of urgency,” he added.
Andrea Fox is Editor-in-Chief of Healthcare IT News.
Email address: afox@himss.org
Healthcare IT News is a publication of HIMSS Media.