Poligage Expert Andy Keiser was quoted in the following Politico article:

BlackBerry Resisted Announcing Major Flaw in Software Powering Cars, Hospital Equipment

By Betsy Woodruff Swan and Eric Geller

A flaw in software made by BlackBerry has left two hundred million cars, along with critical hospital and factory equipment, vulnerable to hackers — and the company opted to keep it secret for months.

On Tuesday, BlackBerry announced that old but still widely used versions of one of its flagship products, an operating system called QNX, contain a vulnerability that could let hackers cripple devices that use it. But other companies affected by the same flaw, dubbed BadAlloc, went public with that news in May.

Two people familiar with discussions between BlackBerry and federal cybersecurity officials, including one government employee, say the company initially denied that BadAlloc impacted its products at all and later resisted making a public announcement, even though it couldn’t identify all of the customers using the software.

The back-and-forth between BlackBerry and the government highlights a major difficulty in fending off cyberattacks on increasingly internet-connected devices ranging from robotic vacuum cleaners to wastewater-plant management systems. When companies such as BlackBerry sell their software to equipment manufacturers, they rarely provide detailed records of the code that goes into the software — leaving hardware makers, their customers and the government in the dark about where the biggest risks lie.

BlackBerry may be best known for making old-school smartphones beloved for their manual keyboards, but in recent years it has become a major supplier of software for industrial equipment, including QNX, which powers everything from factory machinery and medical devices to rail equipment and components on the International Space Station. BadAlloc could give hackers a backdoor into many of these devices, allowing bad actors to commandeer them or disrupt their operations.

Microsoft security researchers announced in April that they’d discovered the vulnerability and found it in a number of companies’ operating systems and software. In May, many of those companies worked with the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency to publicly reveal the flaws and urge users to patch their devices.

BlackBerry wasn’t among them.

Privately, BlackBerry representatives told CISA earlier this year that they didn’t believe BadAlloc had impacted their products, even though CISA had concluded that it did, according to the two people, both of whom spoke anonymously because they were not authorized to discuss the matter publicly. Over the last few months, CISA pushed BlackBerry to accept the bad news, eventually getting them to acknowledge the vulnerability existed.

Then BlackBerry said it didn’t intend to go public to deal with the problem. The company told CISA it planned to reach out privately to its direct customers and warn them about the QNX issue.

Technology companies sometimes prefer private vulnerability disclosures because doing so doesn’t tip off hackers that patching is underway — but also because it limits (or at least delays) any resulting public backlash and financial losses.

But that outreach would only cover a fraction of the affected companies, because BlackBerry also told CISA that it couldn’t identify everyone using its software in order to warn them.

That’s because BlackBerry licenses QNX to “original equipment manufacturers,” which in turn use it to build products and devices for their customers, just as Microsoft sells its Windows operating system to HP, Dell and other computer makers. BlackBerry told the government it doesn’t know where its software ends up, and the people using it don’t know where it came from. Its known customers are a comparatively small group.

“Their initial thought was that they were going to do a private advisory,” said a CISA employee. Over time, though, BlackBerry “realized that there was more benefit to being public.”

The agency produced a PowerPoint presentation, which POLITICO reviewed, stressing that many BlackBerry customers wouldn’t know about the danger unless the federal government or the original equipment manufacturers told them. CISA even cited potential risks to national security and noted that the Defense Department had been involved in finding an acceptable timing for BlackBerry’s announcement.

CISA argued that BlackBerry’s planned approach would leave out many users who could be in real danger. A few weeks ago, BlackBerry agreed to issue a public announcement. On Tuesday, the company published an alert about the vulnerability and urged customers to upgrade their devices to the latest QNX version. CISA issued its own alert as well.

In a statement to POLITICO, BlackBerry did not deny that it initially resisted a public announcement. The company said it maintains “lists of our customers and have actively communicated to those customers regarding this issue.”

“Software patching communications occur directly to our customers,” the company said. “However, we will make adjustments to this process in order to best serve our customers.”

QNX “is used in a wide range of products whose compromise could result in a malicious actor gaining control of highly-sensitive systems,” Eric Goldstein, the head of CISA’s cyber division, said. “While we are not aware of any active exploitation, we encourage users of QNX to review the advisory BlackBerry put out today and implement mitigation measures, including patching systems as quickly as possible.”

Goldstein declined to address CISA’s conversations with BlackBerry but said the agency “works regularly with companies and researchers to disclose vulnerabilities in a timely and responsible manner so that users can take steps to protect their systems.”

Asked about whether the company originally believed QNX was unaffected, Blackberry said its initial investigation into affected software “identified several versions that were affected, but that list of impacted software was incomplete.”

BlackBerry is hardly the first company to disclose a bug in widely used industrial software, and cybersecurity experts say such flaws are to be expected occasionally in highly complex systems. But resolving the QNX problem will be a major task for BlackBerry and the government.

In a June announcement about QNX’s integration into 195 million vehicles, BlackBerry called the operating system “key to the future of the automotive industry” because it provides “a safe, reliable, and secure foundation” for autonomous vehicles. BlackBerry bragged that QNX was the embedded software of choice of 23 of the top 25 electric vehicle makers.

The QNX vulnerability also has the Biden administration scrambling to prevent major fallout. Vulnerabilities in this code could have significant ripple effects across industries — from automotive to health care — that rely heavily on the software. In some cases, upgrading this software will require taking affected devices offline, which could jeopardize business operations.

“By compromising one critical system, [hackers] can potentially hit thousands of actors down that line globally,” said William Loomis, an assistant director at the Atlantic Council’s Cyber Statecraft Initiative. “This is a really clear example of a good return on investment for those actors, which is what makes these attacks so valuable for them.”

After analyzing the industries where QNX was most prevalent, CISA worked with those industries’ regulators to understand the “major players” and warn them to patch the vulnerability, the agency employee said.

Goldstein confirmed that CISA “coordinated with federal agencies overseeing the highest risk sectors to understand the significance of this vulnerability and the importance of remediating it.”

CISA also planned to brief foreign governments about the risks, according to the PowerPoint presentation.

BlackBerry is far from unique in knowing little about what happens to its products after it sells them to its customers, but for industrial software like QNX, that supply-chain blindness can create national security risks.

“Software supply chain security is one of America’s greatest vulnerabilities,” said Andy Keiser, a former top House Intelligence Committee staffer. “As one of the most connected societies on the planet, we remain one of the most vulnerable.”

But rather than expecting vendors to identify all of their customers, security experts say, companies should publish lists of the types of the code included in their software, so customers can check to see if they’re using code that has been found to be vulnerable.

“BlackBerry cannot possibly fully understand the impact of a vulnerability in all cases,” said David Wheeler, a George Mason University computer science professor and director of open source supply chain security at the Linux Foundation, the group that supports the development of the Linux operating system. “We need to focus on helping people understand the software components within their systems, and help them update in a more timely way.”

For years, the Commerce Department’s National Telecommunications and Information Administration has been convening industry representatives to develop the foundation for this kind of digital ingredient list, known as a “software bill of materials.” In July, NTIA published guidance on the minimum elements needed for an SBOM, following a directive from President Joe Biden’s cybersecurity executive order.

Armed with an SBOM, a car maker or medical device manufacturer that learned of a software issue such as the QNX breach could quickly check to see if any of its products were affected.

SBOMs wouldn’t prevent hackers from discovering and exploiting vulnerabilities, and the lists alone cannot tell companies whether a particular flaw actually poses a risk to their particular systems. But these ingredient labels can dramatically speed up the process of patching flaws, especially for companies that have no idea what software undergirds their products.

“Buying software is only the start of the transaction. It is not the end,” said Trey Herr, director of the Atlantic Council’s Cyber Statecraft Initiative.

“It’s not a new problem,” Herr added. “It’s not a problem that’s going away, and what we are doing right now is insufficient for the scale of that problem.”