The Perils of Metrics Misapplied

[Ken White]

A Reuter's OpEd titled "Quantifying the damage of the rush to quantify" highlights the perils of metrics -- vitally important in many cases -- misapplied. The author, David Callahan, emphasizes the propensity for misuse and for cheating that the unending quest for numbers and 'empirical data' creates. As he writes:

"The number-crunching crowd argues that stronger metrics lead to better outcomes, and certainly there are places where this is true."
. . .
"But, as many critics have pointed out, trying to quantify everything is questionable given the subtleties of the human experience."

Many things require metrics, many more can usefully adopt some metrics -- however, a great many things do not lend themselves to applied metrics. As Callahan notes, misuse can drive fudging, outright cheating and some very flawed perceptions. Warfare, particularly in the realm of tactical and operational training, performance and assessment is one place where, in US practice, 'metrics' are overly and wrongly applied. Significantly so...

[Fuchs]

Cheating is a bottom-up problem.
The bigger problem is often (especially in management "controlling") that quantification leads to

* abuse of metrics for putting pressure on subordinates, pressure, pressure, pressure

* abuse of metrics for veiling that the span of control is too wide. Leaders fool themselves by thinking that thanks to metrics they understand what's going on and apply influence (since the numbers change once they took action), but in reality they're as detached from what happens as a monkey playing with a typewriter would be detached from it.

------------
I've always been known for avoiding quantification if possible (not the least because I have often errors in long calculations) and wrote a whole university dissertation without a single equation (still with almost the best possible grade).

There are alternative ways of grasping reality; quantitiative ways and ways depending on a set of experiences, preferences, information that cannot be quantified.
There are nevertheless occasions when operational research calculations can reveal basic and important truths that would have slipped through and not be understood without it. There's also a lot of purpose in applying quantitative methods to very large datasets, as long as you don't expect a high resolution.


There's also something in between; at times it's a good idea to switch from one method to the other simply to shake up things, to break up old misconceptions. It may thus even be a wise move to apply the "inferior" method for a few months or years!

[KenWats]

Ken, a very interesting topic and a good article. In my current job, I get to deal with numbers and metrics on a daily basis and those who like to make "knee jerk" reactions based on the numbers. Numbers are comforting, they provide a certainty that a subjective "better" or "worse" doesn't always give.

The problem is that you're dealing with humans who collect, interpret, the numbers and decide which numbers are meaningful. Humans who (in my world) have pay increases directly related to the numbers they collect. So, there's an active incentive to report things in the best possible light when what's really needed is to see where and how things are going wrong or what else is going wrong because the number is better. Humans who actively manage the system to try to get the best numbers possible- even at the expense of the overall goal (often not realizing the effect on the overall goal).

I'll provide an example from my manufacturing background. "They" decided we needed to get into the 20th Century and measure Inventory Turns (essentially a metric to measure the dollar value of what we have sitting in the warehouse compared to what we ship out in a given month). "They" read in a management book that 9-10 inventory turns is a world class organization and the metric said we were at 6. So, the beatings will continue until inventory turns get up to 9. Well, our product mix was not changed. Customer lead times were not changed (to allow us to actually plan a schedule and run in campaigns of similar products and reduce setup time). The process was not changed or sped up. So, we ended up with a lot less inventory, a customer base used to getting things right away that no longer could, and constant break-in orders in production to keep a customer from running out of material. That's what blind management to numbers gets you from folks who don't get the numbers, process, or customer. I'm sure that staff folks have more military related anecdotes but I think the principles are similar.

Numbers can be incredibly useful. But some thought has to be given to how the numbers are obtained, what incentives you're providing (and are they all good?), what the numbers actually say (and what they don't say). Another fun exercise (probably easier to do in manufacturing than military operations) is to say, "OK, I think the numbers we measured are telling me "X". What else should I expect to see if "X" is really happening? Is that happening too? Or is it something else entirely?".

Just my two cents.

Ken

[ganulv]

As he writes:Many things require metrics, many more can usefully adopt some metrics -- however, a great many things do not lend themselves to applied metrics. As Callahan notes, misuse can drive fudging, outright cheating and some very flawed perceptions.

Thanks, that’s a good one. One day a few years ago as we sat around before class began one of my instructors joined in on a conversation a few of us were having about the GRE. He said that he felt like most members of admissions committees used GRE scores to satisfy their consciences that they had chosen and disqualified the correct people despite the fact that plenty of studies had concluded that higher GRE scores were poor indicators of ultimate success in graduate programs. As GRE scores below a certain point were decent indicators of ultimate failure in graduate programs, he had while head of the department suggested pooling all applicants whom scored above that point (and whom had surpassed a couple of other such indicators) and randomly selecting among them for admission. The suggestion didn’t go through, as one might imagine. But I’ve always liked it as an illustration of how many of us not only can’t deal with randomness but also can’t admit we ourselves act arbitrarily.

[JMA]

A Reuter's OpEd titled "Quantifying the damage of the rush to quantify" highlights the perils of metrics -- vitally important in many cases -- misapplied. The author, David Callahan, emphasizes the propensity for misuse and for cheating that the unending quest for numbers and 'empirical data' creates. As he writes:Many things require metrics, many more can usefully adopt some metrics -- however, a great many things do not lend themselves to applied metrics. As Callahan notes, misuse can drive fudging, outright cheating and some very flawed perceptions. Warfare, particularly in the realm of tactical and operational training, performance and assessment is one place where, in US practice, 'metrics' are overly and wrongly applied. Significantly so...

I have specific experience of this insanity.

When I returned to South Africa after Rhodesia I found the metric nonsense was firmly entrenched (seemingly through Gen Malan who had become contaminated through contact with the US military - staff course and what have you).

The insanity went something like this...

A company of National Servicemen (conscripts) trains for the first year of their service before being to deployed to SWA (South West Africa) for 'border' duty.

On arrival in the operational area they have to pass through a 'training base'. First the whole company is given written exam which relates heavily to operational area SOPS and enemy related info (of which they have hitherto received no training or information). Statistics are produced.

At the end of the week of 'training' (carried out by 'instructors' in their second year of service with little or no operational service) they are retested and guess what?

The improvement is dramatic ... (surprise suprise) ... something like a 30% improvement in the standard! Oh what a wonderful training team they are.

Sound familiar?

[Ken White]

I have specific experience of this insanity.
. . .
Sound familiar?

Too familiar.

Same sort of thing. US Army ROTC Advanced Camp at Bragg, the DCG of then Third Army decided all Cadets should qualify as Sharpshooter or better on the Range. Only available time for refires for those who did not do so was a Sunday. Voila, All Cadets fired Sharpshooter or better on their first try. Happy General, happy NCOs who worked the ranges, happy Cadets *...

Viet Nam. Diktat was promulgated that no body count could be claimed lacking a weapon . Predictably, body count went down, weapons collected count went up. Queries from 'Higher Hq' " You're falling behind. What's the problem down there?" Counts reversed. Amazing...

* The General later wrote an article on leadership in which he cited that incident as an example -- with no trace of irony -- of the 'fact' that you could have perfection if you simply demanded it and obtained metrics to validate performance.

Duty, HONOR, country. Yes...

[Bill Moore]

* The General later wrote an article on leadership in which he cited that incident as an example -- with no trace of irony -- of the 'fact' that you could have perfection if you simply demanded it and obtained metrics to validate performance.

That about sums it up.

[Firn]

There once was a business owner who was interviewing people for a division manager position. He decided to hire person who could answer the question "How much is 2+2?"

The engineer pulled out his slide rule and shuffled it back and forth, and finally announced, "It lies between 3.98 and 4.02."

The mathematician said, "In two hours I can demonstrate it equals 4 with the following short proof."

The social worker said, "I don't know the answer, but I a glad that we discussed this important question.

The attorney stated, "In the case of Svenson vs. the State, 2+2 was declared to be 4."

The trader asked, "Are you buying or selling?"

The accountant looked at the business owner for a moment. He got out of his chair, went to see if anyone was listening at the door, then went to the window and pulled the drapes closed.

Returning to the business owner, the accountant slowly leaned across the desk and said in a low voice:

"What would you like it to be?"

In business metrics are very important but they should always be seen as help to understand the means and to reach the goals, not as the mean and the goal itself bar the very basic ones as ROI, ROCE etc. Accounting & Controlling are surprisingly interesting fields with often most revealing and shocking findings. I remember how a board of a European company in which I wanted to invest quite some money offered an interest free 1 million € loan to it's members backed by the value of the convertible bond for which that offer was reserved. Getting the whole bond on less then 30 cents on the € was quite a business with that loaned money, considering the bonds shot up to 110 cents a year later and hover still above the 100 cents. Nice but of course like with options that loan was not listed among the expenses... Shareholder money is clearly interest-free and dilution is an imagined problem


In the military training the proper use of metrics should be much more difficult to implement then in business. The big problem of a heavy reliance are the big incentives to game the system, training for the score and 'making it work' to earn the benefits.

[JMA]

In business metrics are very important but they should always be seen as help to understand the means and to reach the goals, not as the mean and the goal itself bar the very basic ones as ROI, ROCE etc. Accounting & Controlling are surprisingly interesting fields with often most revealing and shocking findings. I remember how a board of a European company in which I wanted to invest quite some money offered an interest free 1 million € loan to it's members backed by the value of the convertible bond for which that offer was reserved. Getting the whole bond on less then 30 cents on the € was quite a business with that loaned money, considering the bonds shot up to 110 cents a year later and hover still above the 100 cents. Nice but of course like with options that loan was not listed among the expenses... Shareholder money is clearly interest-free and dilution is an imagined problem


In the military training the proper use of metrics should be much more difficult to implement then in business. The big problem of a heavy reliance are the big incentives to game the system, training for the score and 'making it work' to earn the benefits.

Business? Its a different world. One which needs to be tolerated and endured rather than savoured. Sydney Jary in the magnificent book '18 Platoon' explains the difference and why good military leaders have a problem adjusting to the 'false' world of business.

Let us now leave the service ethos, where leadership is appreciated and expected of those in authority. I went into business where I found an almost total absence of leadership qualities. There were only two exceptions, one a retired Brigadier REME, the other a Swiss machine tool manufacturer.

To succeed in business other characteristics - I will not dignify by calling them qualities - are required. To succeed you require energy, ambition, enthusiasm, plausibility and a degree of intolerant ruthlessness. A cocktail of egocentric characteristics which hardly inspires the trust of subordinates. It does, however, gain the confidence of clients. It sells.

As a final note on this... had I known when I was a young officer that I was fighting for (and soldiers were dying for) the protection of the assets and profits of the ethically and morally deprived we call 'the captains of industry' I would not have done it.

[Surferbeetle]

Business? Its a different world. One which needs to be tolerated and endured rather than savoured.

BS

Good business is about figuring out how to make cost effective and efficient contributions towards the greater good. Done correctly lives are enriched, resources are appropriately prioritized and allocated, and we all move forward. Done incorrectly all of the foibles of the human animal are amplified and we collectively descend back into the muck and mire from whence we all sprang. Transparency and accountability helps to keep things trending towards good business....

With respect to metrics, they are essential, as is keeping firmly in mind that moderation in all things is necessary.

[JMA]

BS

Good business is about figuring out how to make cost effective and efficient contributions towards the greater good. Done correctly lives are enriched, resources are appropriately prioritized and allocated, and we all move forward. Done incorrectly all of the foibles of the human animal are amplified and we collectively descend back into the muck and mire from whence we all sprang. Transparency and accountability helps to keep things trending towards good business....

With respect to metrics, they are essential, as is keeping firmly in mind that moderation in all things is necessary.

And there I was holding my breath waiting for examples of 'good business'. Silly me.

[Surferbeetle]

And there I was holding my breath waiting for examples of 'good business'. Silly me.

Silly you? No, that's not my assessment

Examples? How about South African Infrastructure for starters?

SA's R800bn infrastructure drive, 25 October 2011, http://www.southafrica.info/business...udget2011b.htm

South Africa is set to spend over R802-billion on infrastructure development over the next three years as part of a government drive to ease bottlenecks and reduce costs in the economy, "crowd in" private investment and improve access to export markets.

Presenting his Medium Term Budget Policy Statement in Parliament on Monday, Gordhan said public-sector investment in infrastructure - which has increased from 4.3% of the country's gross domestic product (GDP) in 2005 to 7.5% in the first half of 2011 - remained central to the government's economic development plans.

"Infrastructure projects in energy, roads, rail, telecommunications and water will ease bottlenecks and reduce costs in the rest of the economy, crowding in private investment and improving access to export markets," Gordhan told parliamentarians in Cape Town.

[JMA]

Silly you? No, that's not my assessment

Examples? How about South African Infrastructure for starters?

SA's R800bn infrastructure drive, 25 October 2011, http://www.southafrica.info/business...udget2011b.htm

No that is not business. Nothing to do with business other than (like the Arms Deal of some years ago) for the 'lucky' winners of the tenders to do the work and those political insiders who will stand to cream off R80-100 billion in terms of kickbacks and 'commissions' who will be smiling.

Would be interested to hear about an example of 'good business' as per your definition.

[Stan]

Never in my life have I seen such foolishness with our federal budget. From the old lady in tennis shoes that destroys your travel voucher in hours to save .99 cents (that you in fact paid) to the "lowest bidder to a beltway bandit" with the cheapest widget with the highest overhead costs.

If we even remotely followed some small business savvy we would never have entered into a war and would doubtfully be in debt.

[Bob's World]

Sometimes we measure the wrong things because we reasonably believe them to be the right things. Perhaps we need a "metric metric" that measures if a metric is of any value...

But sometimes we measure the wrong thing because it tells the story we want to tell. A recent example: Last week I visited a major defense think tank in Washington DC. They had recently published a perspective on military budgets and had determined that "SOCOM was a big winner" and wanted to gain our insights on their analysis.

Detailed analysis of military budgets during and following conflict from WWII forward showed a fairly standard degree of rise and fall as compared against Gross Domestic Product (GDP). The standard post-conflict budget throughout that era was 3% of GDP.

The recommendation was that the US was very much in a situation as faced by President Eisenhower and should look at making similar decisions regarding how to invest a budget based on 3% of GDP. All quite reasonable.

My question was: "Why do you only look at military budgets post-WWII? We were in a Cold War during that period that we are not in now. During the Cold War the US adopted the geo-strategic realities of a European continental power, demanding a large standing army in addition to other capacities designed specifically to deter the Soviets. The geo-strategic reality of the US is one of a maritime nation that has no such requirement for such a peacetime ground force. Today we are back to our geo-strategic roots, and arguably the period of time from 1900 -1941 is far more applicable to our current situation than the period of 1947-2012. Did you look at that era as well?

Answer: "Yes"

Question: "Really, and what was the % of GDP applied to defense in the post-conflict periods of that era"?

Answer: "1 percent"

Sometimes measuring the right thing doesn't tell the story you want to hear, so you refine what you measure until you finally get the answer you want. What I found particularly disturbing was that they had done the research, had the answer, and had then deliberately crafted the findings to tell the story they thought the customers would want to hear. All their facts were true and accurate as far as I could tell, they just didn't tell the whole story as it presented inconvenient results.

Now, as to what defense spending should be, it may not be 1% given the advent of expensive new domains such as air, space and cyber; but I doubt it is 3% unless we cling to a force we want, rather than build the force we need. So my takeaway is that we probably can, and should take defense cuts down to 2% of GDP. After all, this is not about SOF or any of the Services "winning," this is about the United States of America "winning." (in quotes, as one must not only define the right level to assess "winning" at, but also get to the right definition of what constitutes a win.)

[Fuchs]

Imagine the U.S: was only half as rich. Would this mean the %GDP budget should be 4%, not 2%?

Why is the GDP a good base for determining a budget at all? To me it looks rather like a variable that tells you something about affordability than about necessity.


You could for example look at Purchasing Power Parities, military spending and inefficiency levers (strategic logistics, degree of bureaucratic inefficiency) in one bloc and compare this with your bloc (say, PRC + NK vs. Taiwan+U.S.).
Then you have an idea about the relative power situation and can continue with thoughts about wargames and such.

[Bob's World]

I think this goes more in the up direction than down: We have so much military because we can afford it, not because we need it. "Wants" are often sold as "needs" in this game.

Where is the perfect balance between what one can afford and what one needs? There is no perfect metric to get there. I would argue that most European powers have the militaries they can get away with due to alliances with a US that has too large of a ground force for its own needs. Currently we see Secretary Panetta having to mollify European allies who are concerned they may have to resource a fairer share of their own defense.

http://www.nytimes.com/2012/02/05/wo...rope.html?_r=1

GDP over history tells an important story; particularly in a democracy where the people have the ultimate say on such things (though like the Titanic, we are big engine, no brakes, and small rudder, and think we are unsinkable), so such adjustments take time. It shows tolerance for defense spending. The results of the effectiveness of US military power speak for themselves.

Many love to wheel out tired arguments of how we were "unprepared" for both World Wars and Korea; though were the decisive ground force in all three. Being "prepared" has enabled US officials to jump into all kinds of small to medium conflicts that we would have been forced to employ other options on if we had not possessed a warfighting army sitting on the shelf ready to go. I'm not convinced that is better.

(oh, and the "additional $500B" in cuts that SecDef calls "crazy"? I suspect will seem less crazy after the election)

[Entropy]

Why is the GDP a good base for determining a budget at all? To me it looks rather like a variable that tells you something about affordability than about necessity.

Another imperfect metric! Personally, I like defense spending per capita. Reducing our spending by 25-50% would put on on par with Europe and that would give us an idea of what our capabilities would be with that level of spending.

Today we are back to our geo-strategic roots, and arguably the period of time from 1900 -1941 is far more applicable to our current situation than the period of 1947-2012. Did you look at that era as well?

Except we still have our Cold War defensive alliances. As long as we're obligated to defend allies around the world, then we'll need the capability to project power and deploy combat forces to those areas. That's not cheap.

Also, it's putting the cart before the horse to simply declare that we only "need" some amount of defense spending. Can we have the capabilities we have currently at 1% of GDP? No. What will 1% get us and what will our alliances and foreign policy look like if spending was restricted to that amount?

[Fuchs]

Except we still have our Cold War defensive alliances. As long as we're obligated to defend allies around the world, then we'll need the capability to project power and deploy combat forces to those areas. That's not cheap.

Read the treaties. The North Atlantic Treaty for one does not mandate a specific military strength. In fact, it doesn't even mandate a specific reaction to an invasion. There's a lot of latitude involved (The Lisbon treaty is on paper a much stricter alliance, but almost nobody knows about this function!).

[Bob's World]

Read the treaties. The North Atlantic Treaty for one does not mandate a specific military strength. In fact, it doesn't even mandate a specific reaction to an invasion. There's a lot of latitude involved (The Lisbon treaty is on paper a much stricter alliance, but almost nobody knows about this function!).

The Cold War is over. Spending too much on defense weakens us just as much as spending too little. Perhaps even more so.

Are we stronger when we borrow Billions from China to produce military capabilities designed to contain China??? No. China understands that and plays our fears masterfully.

And just because we have clung to outdated alliances along with outdated perspectives on how much military capacity we need to have on the shelf does not mean we are somehow duty bound forever to protect them from whatever might come along.

Our duty is to our own people and our vital interests. As a powerful maritime nation, like Britain before us, we have the luxury of being able to focus on our interests, and employ our wealth to play a counter balance by changing teams and providing resources to others to fight their own fights as it suits our interests best on a case by case basis. This whole "Red Team, Blue Team" mentality from the Cold War is detrimental to our security, as it locks us into bad partnerships and prevents us from forming smart ones.