Microsoft Windows powers more than a billion PCs and millions of servers worldwide, many of them playing key roles in facilities that serve customers directly. So, what happens when a trusted software provider delivers an update that causes those PCs to immediately stop working? As of July 19, 2024, we know the answer to that question: Chaos ensues. In this case, the trusted software developer is a firm called CrowdStrike Holdings, whose previous claim to fame was being the security firm that analyzed the 2016 hack of servers owned by the Democratic National Committee. That's just a quaint memory now, as the firm will forever be known as The Company That Caused The Largest IT Outage In History. It grounded airplanes, cut off access to some banking systems, disrupted major health care networks, and threw at least one news network off the air. Microsoft estimates that the CrowdStrike update affected 8.5 million Windows devices. That's a tiny percentage of the worldwide installed base, but as David Weston, Microsoft's Vice President for Enterprise and OS Security, notes, "the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services." According to a Reuters report, "Over half of Fortune 500 companies and many government bodies such as the top US cybersecurity agency itself, the Cybersecurity and Infrastructure Security Agency, use the company's software." CrowdStrike, which sells security software designed to keep systems safe from external attacks, pushed a faulty "sensor configuration update" to the millions of PCs worldwide running its Falcon Sensor software. That update was, according to CrowdStrike, a "Channel File" whose function was to identify newly observed, malicious activity by cyberattackers. Although the update file had a .sys extension, it was not itself a kernel driver. It communicates with other components in the Falcon sensor that run in the same space as the Windows kernel, the most privileged level on a Windows PC, where they interact directly with memory and hardware. CrowdStrike says a "logic error" in that code caused Windows PCs and servers to crash within seconds after they booted up, displaying a STOP error, more colloquially known as the Blue Screen of Death (BSOD). Repairing the damage from a flaw like this is a painfully tedious process that requires manually rebooting every affected PC into the Windows Recovery Environment and then deleting the defective file from the PC using the old-school command line interface. If the PC in question has its system drive protected by Microsoft's BitLocker encryption software, as virtually all business PCs do, the fix requires one extra step: entering a unique 48-character BitLocker recovery key to gain access to the drive and allow the removal of the faulty CrowdStrike driver. In that case, McAfee had delivered a faulty virus definition (DAT) file to PCs running Windows XP. That file falsely detected a crucial Windows system file, Svchost.exe, as a virus and deleted it. The result, according to a contemporary report, is that "affected systems will enter a reboot loop and [lose] all network access." The parallels between that 2010 incident and this year's CrowdStrike outage are uncanny. At its core was a defective update, pushed to millions of PCs running a powerful software agent, causing the affected devices to stop working. Recovery required manual intervention on every single device. Plus, the flawed code was pushed out by a public security company desperately trying to grow in a brutally competitive marketplace. Less than a month earlier, according to a report from The Stack, CrowdStrike released a detection logic update for the Falcon sensor that exposed a bug in the sensor's Memory Scanning feature. "The result of the bug," CrowdStrike wrote in a customer advisory, "is a logic error in the CsFalconService that can cause the Falcon sensor for Windows to consume 100% of a single CPU core." The company rolled back the update, and customers were able to resume normal operations by rebooting. At the time, computer security expert Will Thomas noted on X/Twitter, "[T]his just goes to show how important it is to download new updates to one machine to test it first before rolling out to the whole fleet!" In that 2010 incident, the root cause turned out to be a complete breakdown of the QA process. It seems self-evident that a similar failure in QA is at work here. Were these two CrowdStrike updates not tested before they were pushed out to millions of devices? Part of the problem might be a company culture that's long on tough talk. In the most recent CrowdStrike earnings call, CEO George Kurtz boasted about the company's ability to "ship game-changing products at a rapid pace," taking special aim at Microsoft: Complete details are posted on OUR FORUM.

Timing is everything—and that’s especially true for the millions of Microsoft Windows users with a fast-approaching July 4 deadline to update their systems. It’s just two weeks ago that we saw a patched Windows vulnerability come back to life. While Microsoft had suggested no known exploits for CVE-2024-26169, Symantec’s security researchers thought somewhat differently, with “some evidence” that attackers “compiled a CVE-2024-26169 exploit prior to patching.” And it’s just last month that several U.S. government agencies—including CISA and the FBI—collaborated on a Cybersecurity Advisory warning that “Black Basta affiliates have impacted a wide range of businesses and critical infrastructure in North America, Europe, and Australia. As of May 2024, Black Basta affiliates have impacted over 500 organizations globally.” Black Basta is a Ransomware-as-a-Service (RaaS) group that has targeted “12 out of 16 critical infrastructure sectors,” the agencies said, “including the Healthcare and Public Health (HPH) Sector.” But the group’s activities have extended well beyond the public sector, hitting the likes of Hyundai, Rheinmetall, Capita and ABB. Timing is everything. And these stories come together—somewhat awkwardly for Microsoft—because Symantec suggested it was “the Cardinal cybercrime group (aka Storm-1811, UNC4393), which operates the Black Basta ransomware” that was likely exploiting the privilege escalation vulnerability in Microsoft’s Windows Error Reporting Service for several weeks before it was patched in March. CISA has added CVE-2024-26169 to its Known Exploit Vulnerability (KEV) catalog, flagging that it is “known to be used in ransomware campaigns” and mandating all Windows systems be updated or shut down by July 4. That mandate only applies to US federal agencies, but CISA says it “strongly urges all organizations to reduce their exposure to cyberattacks by prioritizing timely remediation.” Black Basta has now generated significantly more than $100 million in ransomware payments, and so leaving Windows systems unpatched is a gamble no organization should take. All should follow CISA’s July 4 update mandate. While the specific issue here is less relevant to personal users, update right away if you haven’t done so. Since this article was published, the situation for Windows 11 users has been complicated by spreading news of an unexpected restart loop impacting some users installing June’s Windows 11 KB5039302. Microsoft has warned users that “after installing updates released June 26, 2024 (KB5039302), some devices might fail to start,” advising that “affected systems might restart repeatedly and require recovery operations in order to restore normal use.” KB5039302 is not a mandatory update in itself and is not a security update—as such, put those headlines aside and proceed as normal. Do not confuse this update with the security patching that resolves the Microsoft Windows Error Reporting Service vulnerability. In any case, it’s highly likely that your Windows 11 PC will be unaffected by the new issue. This restart loop impacts enterprise machines running “virtual machines tools and nested virtualization features,” Microsoft has said, which means home users are less likely to be hit. Users will still see relevant updates as available. The issues covered by CISA’s warning were patched ahead of June’s release, and given the Black Basta angle, the urgency remains. And that means that while Microsoft may have pulled KB5039302 for some users, you should still ensure you update your PC ahead of the July 4 deadline. The much bigger issue that does impact Windows home users is now fast approaching, albeit that deadline is still more than a year away on October 14, 2025. Just days before Symantec’s report, we saw Microsoft again urging Windows 10 users to upgrade to Windows 11. With a daunting 70% of users yet to make the switch ahead of next year’s end-of-life, that challenge is becoming ever more acute and Microsoft’s nags have started to hit PCs worldwide. Learn more by visiting OUR FORUM.