Sunday, March 25, 2007

How not to do a cost-benefit analysis

This GAO report could be a primer in what not to do when comparing costs and benefits. It is a report on the efforts of the Department of Homeland Security (DHS) to upgrade their supply of radiation detection monitors as part of the effort to protect us against the smuggling of nuclear material.

DHS has decided to upgrade the current generation of monitors and has sponsored R&D and testing activities to develop a new monitor. One problem is the estimated cost of the new monitors - six times that of the current monitors. So, the question DHS was asked to answer was: is the increased cost worth it?

Problem 1 - They had tested the new monitors a couple of years ago. But, in doing this analysis they decided not to use the results of these tests. They just assumed that the new monitors would detect highly enriched uranium 95% of the time. This assumption is just that - an assumption. It has yet to be proven.

Problem 2 - They tested the current monitors using data from tests conducted in 2004. These tests were unreliable, or so said the people who conducted the tests in 2004.

Problem 3 - DHS tested only for highly enriched uranium. There are several other materials that are as dangerous as uranium.

Problem 4 - The price of the current monitors is $55,000. DHS assumed a price of $131,000 in their analysis.

Problem 5 - The new monitors will reduce unnecessary secondary inspections, or so it is thought. However, the monetary value of this feature is unknown as they don't know the cost of these secondary inspections.

Problem 6 - There was no attempt to estimate life cycle costs.

Makes you feel good. Doesn't it?

No comments: