Friday, January 28, 2005

Metrics for scientific excellence

From the 1/28/2005 LANL NewsBulletin:

Jan. 26, 2005

Metrics for scientific excellence

Several of the recent letters to the Reader’s Forum have commented upon the pretentious nature of the Laboratory’s motto “The World’s Greatest Science ....” It does appear that the motto is yet another example of upper management’s wishful thinking that saying it makes it real. But just what are the metrics for great science? If great science is the Lab’s product, how is it measured? Why should it be measured?

Developing metrics for scientific output is of critical importance to the future of the Lab. In industrial and commercial settings, ‘output’ is quantifiable – widgets produced, gross and net income, growth rate, etc. Thus, ‘cost’ metrics like accident rate, lost work days, production and compliance costs can be ratioed to ‘output,’ allowing one to determine when a change has a net positive or negative impact. That is the fundamental premise of benefit/cost analysis. Without a metric for output (great science), ‘cost metrics’ at the Lab, like accident rate, security incidents and financial accountability, get ratioed against each other in a spiraling closed loop that is totally divorced from output. Thus, there is no way to quantify benefit/cost. And no way to determine when increasing formality of operations has moved from a positive territory into negligible increased benefit into the negative regime. I assert that the increased formality of operations protocol has already moved us into negative benefit territory (as illustrated by the uptick in safety and security incidents), but that’s another issue...

In his message to the National Nuclear Security Administration (as reported in the Los Alamos Monitor “Domenici addresses lab's RFP concerns,” Sen. Pete Domenici targets several key areas of deficiency in the draft request for proposal. But he doesn’t go far enough.

Unless we can determine just what is the desired output from the Lab and how to quantify it, all those other metrics become meaningless and we will become another Rocky Flats. And there are no metrics for great science in the RFP. Admittedly, it’s a complex issue, where ‘output’ varies from project to project. But certainly some metrics like number of manuscripts, number of heat sources, number of pits, active beam time, active user-facility days, increase in customer base, etc. can be developed to require that the contractor actually support the scientific output. That is the purpose of having a national laboratory in the first place instead of just continuing down the current path of ignoring the impact of various arbitrary directives upon our ability to do ‘great science.’

--Mary Barr

LANL leaders have chosen the R&D 100 list as a metric for success. No one has questioned this figure of merit. The credibility of the listing can easily be measured by comparing the licensing fees from the accumulated LANL patents with fees collected by other labs.
The lab has patent attorneys, an intellectual property office and publicists who help the scientist compete for these R&D 100 awards. Those offices alone cost on the order of $10,000,000/year. I have been told that the total fees collected by LANL amount to about $100,000 per year. Other labs with smaller research budgets do very much better. What’s wrong?
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?