Predictive analytics faces skepticism in government
When the Homeland Security Department decided it would roll out a predictive analytics capability for cybersecurity in a bid to advise other government agencies on risk mitigation priorities, it ran into an obstacle, said Tom Finan, DHS senior cybersecurity strategist and counsel.
The response he got from intended audiences was "when somebody says 'predictive analytics' to us, we think about fortune tellers and crystal balls--and we don't have crystal balls," Finan said Sept. 17 during a session of the Predictive Analytics World-Government conference in Washington, D.C.
DHS officials also found that examining the motivation of cyber attackers didn't lead to strong predictive cases, since those motivations vary widely and are subject to change--or, in a word, are unpredictable.
As a result of those pressures, DHS shifted heavily to consequence analysis, Finan said.
"If we can show a picture of how a cascading infrastructure failure is going to make a particular locale economically unviable, that'll have a lot more power than saying, 'Hey, here's a predictive product, will you read it and make a decision?'" Finan said.
The department can be sure it avoids recommending a suboptimization of risk mitigation dollars by leaving it up to local government agencies to determine priority for themselves, Finan added.
DHS "is never going to be able to get in the particular facts and scenarios of every single jurisdiction in the analysis," he said.
When using predictive analytics to construct response plans, it's important not to construct scenarios too tightly, warned David Kaufman, Federal Emergency Management Agency director of policy and program analysis, who also spoke at the conference panel.
"We rely too much on static methodologies for a world that's dynamic," he said. Often there's a tendency to plaster over uncertainty, but that results in brittleness. Kaufman cited FEMA predictive modeling for a pandemic that "we had to chuck most of out the window" during the 2009 H1N1 flu wave because despite 2 years of "disciplined, high-level pandemic planning," FEMA has only assumed that the pandemic would originate in the Far East.
"Our plans were not aligned, or not flexible enough, when the origination point was below us," he added.