The Department of Homeland Security is using the wrong mathematic approach to assess the risk of terrorist attacks, which means that “only low confidence should be placed in most of the risk analyses conducted by DHS,” according to an independent study.
DHS uses these quantitative analyses to guide the spending of its $55 billion budget and to decide if it’s more cost-effective to, say, install anthrax sanitizers in every post office or to park a bomb sniffing dog at every airport.
A committee within the National Academy of Sciences — the scientific advisory organization chartered by Congress — found DHS does a good job forecasting how much risk natural disasters pose to certain areas and the likelihood of random catastrophes, like power plants exploding or planes crashing. But DHS also uses the same techniques to assess the risk of human terrorist attacks, an approach that is “fatally flawed,” says Tony Cox, a consultant and an editor for Risk Analysis: An International Journal.
“The problem with attacks by Al Qaeda or other intelligent attackers is they are not random events,” Cox told the Center for Public Integrity. “They are decisions based on information and goals and calculations about the best way to reach those goals.”
A DHS spokesman did not immediately respond to a request for comment about the report.
Game theory a better tool
The National Academy of Sciences panel’s core recommendation: DHS should move away from “probabilistic risk analysis” for terror assessments and instead use other decision tools, like game theory. “The committee saw little evidence of DHS being aware of the full range of state-of-the art risk analysis and decision support tools, including those designed specifically to deal with deep uncertainty,” the report said.
The probabilistic risk analysis approach works well in situations where there is a lot of data, like weather disasters or failures in complex engineered systems. It feeds data on how often different parts of a system fail into a mathematical model that calculates how often the whole system fails, and how severe the resulting losses are likely to be.
With terrorism, DHS is multiplying the mathematical expressions for threat, vulnerability and consequences of an event to determine the overall risk. The report found the department’s formula “seriously deficient” and its estimates for the variables “crude, simplistic, and misleading.”
Vicki Bier, a professor at the University of Wisconsin in Madison who advised the National Academy of Sciences panel in the early phases of its 15-month study, points out that the DHS threat assessments “tend to assume that the other side wants to maximize” deaths or property damage. But, she added, “they may be perfectly happy to launch high profile attacks that get lots of media coverage.”
2008 warning
This is not the first time DHS has received this kind of warning.
In 2008, the National Research Council examined DHS’s risk assessment model for biological threats and flatly declared it “should not be used as a basis for decision making” to assess the risk of biological, chemical, or radioactive threats. The 2010 report says since then, updating those methods has been “incremental, and a much deeper change is necessary.”
How DHS deals with uncertainty especially concerned the science panel. Instead of disclosing how confident the department is in an assessment, DHS downplays them, which can give a false impression of precision in its estimates. Furthermore, it hobbles DHS’s ability to make good decisions about future data collection. “This is not an acceptable way of dealing with uncertainty,” the report says.
Even if the DHS can shore up the technical aspects of its methodology, Cox worries that institutional inertia may be difficult and expensive to overcome. Reporting formats, grant protocols, budget decisions and work flows have been built on the vulnerability-threat-consequence framework.
“Are they wedded to a conceptual framework that they can’t wean themselves from?” asks Cox, who worked on the 2008 report. “It may be almost impossible for people in the trenches to stop using it. So the fact that it’s demonstrably ineffective may not be enough to change the tremendous momentum.”
UPDATE — 9/20/10: After publication, DHS Spokesman Bobby Whithorne said, “DHS has updated and strengthened its risk analysis efforts — addressing many of the issues raised in the report — in the two years since the National Academies began its review.”
FAST FACT: The National Academy of Sciences panel that produced the 148-page report was led by John Ahearne, a nuclear weapons expert and former chairman of the U.S. Nuclear Regulatory Commission. Ahearne is also an adjunct engineering professor at Duke and a former president of the Society for Risk Analysis.
Other new reports released by agencies including the Government Accountability Office (GAO) and various federal Offices of Inspector General (OIG):
FINANCE
* IRS must set deadlines to phase out use of taxpayers’ Social Security numbers on agency correspondence to prevent identity theft (OIG)
* IRS lawyers should issue within 120 days private letter rulings requested by companies and publish more guidance that would reduce the need for private rulings (OIG)
ENVIRONMENT
* U.S. Capitol Architect must ask Congress for legal authority to build battery recharging stations on the Capitol grounds for hybrid cars owned by lawmakers or their employees (GAO)
* EPA needs to revise any outdated or inconsistent agreements it has with states to regulate pollution under the Clean Water Act (OIG)
* NASA Administrator Charles Bolden’s April phone call with a Marathon Oil Corp. official while considering an alternative fuel development project did not violate federal laws. But it did raise the appearance of a conflict of interest and violated the Obama administration’s ethics pledge for all appointees because Bolden owns more than $500,000 worth of Marathon stock (OIG)
HEALTH CARE
* Medicare can save money if it develops more accurate method to estimate prices of drugs to treat end stage renal disease instead of using the Producer Price Index (OIG)
* Medicare plans Oct. 5 meeting on health reform law provision encouraging providers to create integrated health care delivery systems that will cut costs, test new reimbursement methods (OIG)
NATIONAL SECURITY
* Homeland Security Dept. needs more staff to oversee Boeing Co’s work on the Secure Border Initiative, a $7.6 billion electronic fence along the U.S.-Mexico border (OIG)
MISC.
* U.S. tsunami forecasting centers in Hawaii and Alaska need better coordination with emergency managers, news media, and the public (National Research Council)
* Virgin Island Port Authority, which spent $211.7 million on contracts in past decade, is losing tax revenue because of sloppy records and weak financial reporting (OIG)
* District of Columbia public school system wants stimulus money for its Head Start program for kids but it is unable to “manage and account for federal funds” (OIG)