Results 1 to 20 of 30

Thread: The Perils of Metrics Misapplied

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Council Member Ken White's Avatar
    Join Date
    May 2007
    Location
    Florida
    Posts
    8,060

    Default The Perils of Metrics Misapplied

    A Reuter's OpEd titled "Quantifying the damage of the rush to quantify" highlights the perils of metrics -- vitally important in many cases -- misapplied. The author, David Callahan, emphasizes the propensity for misuse and for cheating that the unending quest for numbers and 'empirical data' creates. As he writes:
    "The number-crunching crowd argues that stronger metrics lead to better outcomes, and certainly there are places where this is true."
    . . .
    "But, as many critics have pointed out, trying to quantify everything is questionable given the subtleties of the human experience."
    Many things require metrics, many more can usefully adopt some metrics -- however, a great many things do not lend themselves to applied metrics. As Callahan notes, misuse can drive fudging, outright cheating and some very flawed perceptions. Warfare, particularly in the realm of tactical and operational training, performance and assessment is one place where, in US practice, 'metrics' are overly and wrongly applied. Significantly so...

  2. #2
    Council Member Fuchs's Avatar
    Join Date
    May 2008
    Posts
    3,189

    Default

    Cheating is a bottom-up problem.
    The bigger problem is often (especially in management "controlling") that quantification leads to

    * abuse of metrics for putting pressure on subordinates, pressure, pressure, pressure

    * abuse of metrics for veiling that the span of control is too wide. Leaders fool themselves by thinking that thanks to metrics they understand what's going on and apply influence (since the numbers change once they took action), but in reality they're as detached from what happens as a monkey playing with a typewriter would be detached from it.

    ------------
    I've always been known for avoiding quantification if possible (not the least because I have often errors in long calculations) and wrote a whole university dissertation without a single equation (still with almost the best possible grade).

    There are alternative ways of grasping reality; quantitiative ways and ways depending on a set of experiences, preferences, information that cannot be quantified.
    There are nevertheless occasions when operational research calculations can reveal basic and important truths that would have slipped through and not be understood without it. There's also a lot of purpose in applying quantitative methods to very large datasets, as long as you don't expect a high resolution.


    There's also something in between; at times it's a good idea to switch from one method to the other simply to shake up things, to break up old misconceptions. It may thus even be a wise move to apply the "inferior" method for a few months or years!

  3. #3
    Council Member
    Join Date
    Apr 2010
    Location
    Northern New Jersey
    Posts
    40

    Default

    Ken, a very interesting topic and a good article. In my current job, I get to deal with numbers and metrics on a daily basis and those who like to make "knee jerk" reactions based on the numbers. Numbers are comforting, they provide a certainty that a subjective "better" or "worse" doesn't always give.

    The problem is that you're dealing with humans who collect, interpret, the numbers and decide which numbers are meaningful. Humans who (in my world) have pay increases directly related to the numbers they collect. So, there's an active incentive to report things in the best possible light when what's really needed is to see where and how things are going wrong or what else is going wrong because the number is better. Humans who actively manage the system to try to get the best numbers possible- even at the expense of the overall goal (often not realizing the effect on the overall goal).

    I'll provide an example from my manufacturing background. "They" decided we needed to get into the 20th Century and measure Inventory Turns (essentially a metric to measure the dollar value of what we have sitting in the warehouse compared to what we ship out in a given month). "They" read in a management book that 9-10 inventory turns is a world class organization and the metric said we were at 6. So, the beatings will continue until inventory turns get up to 9. Well, our product mix was not changed. Customer lead times were not changed (to allow us to actually plan a schedule and run in campaigns of similar products and reduce setup time). The process was not changed or sped up. So, we ended up with a lot less inventory, a customer base used to getting things right away that no longer could, and constant break-in orders in production to keep a customer from running out of material. That's what blind management to numbers gets you from folks who don't get the numbers, process, or customer. I'm sure that staff folks have more military related anecdotes but I think the principles are similar.

    Numbers can be incredibly useful. But some thought has to be given to how the numbers are obtained, what incentives you're providing (and are they all good?), what the numbers actually say (and what they don't say). Another fun exercise (probably easier to do in manufacturing than military operations) is to say, "OK, I think the numbers we measured are telling me "X". What else should I expect to see if "X" is really happening? Is that happening too? Or is it something else entirely?".

    Just my two cents.

    Ken

  4. #4
    Council Member ganulv's Avatar
    Join Date
    Dec 2010
    Location
    Berkshire County, Mass.
    Posts
    896

    Default

    Quote Originally Posted by Ken White View Post
    As he writes:Many things require metrics, many more can usefully adopt some metrics -- however, a great many things do not lend themselves to applied metrics. As Callahan notes, misuse can drive fudging, outright cheating and some very flawed perceptions.
    Thanks, that’s a good one. One day a few years ago as we sat around before class began one of my instructors joined in on a conversation a few of us were having about the GRE. He said that he felt like most members of admissions committees used GRE scores to satisfy their consciences that they had chosen and disqualified the correct people despite the fact that plenty of studies had concluded that higher GRE scores were poor indicators of ultimate success in graduate programs. As GRE scores below a certain point were decent indicators of ultimate failure in graduate programs, he had while head of the department suggested pooling all applicants whom scored above that point (and whom had surpassed a couple of other such indicators) and randomly selecting among them for admission. The suggestion didn’t go through, as one might imagine. But I’ve always liked it as an illustration of how many of us not only can’t deal with randomness but also can’t admit we ourselves act arbitrarily.
    If you don’t read the newspaper, you are uninformed; if you do read the newspaper, you are misinformed. – Mark Twain (attributed)

  5. #5
    Banned
    Join Date
    Mar 2010
    Location
    Durban, South Africa
    Posts
    3,902

    Default

    Quote Originally Posted by Ken White View Post
    A Reuter's OpEd titled "Quantifying the damage of the rush to quantify" highlights the perils of metrics -- vitally important in many cases -- misapplied. The author, David Callahan, emphasizes the propensity for misuse and for cheating that the unending quest for numbers and 'empirical data' creates. As he writes:Many things require metrics, many more can usefully adopt some metrics -- however, a great many things do not lend themselves to applied metrics. As Callahan notes, misuse can drive fudging, outright cheating and some very flawed perceptions. Warfare, particularly in the realm of tactical and operational training, performance and assessment is one place where, in US practice, 'metrics' are overly and wrongly applied. Significantly so...
    I have specific experience of this insanity.

    When I returned to South Africa after Rhodesia I found the metric nonsense was firmly entrenched (seemingly through Gen Malan who had become contaminated through contact with the US military - staff course and what have you).

    The insanity went something like this...

    A company of National Servicemen (conscripts) trains for the first year of their service before being to deployed to SWA (South West Africa) for 'border' duty.

    On arrival in the operational area they have to pass through a 'training base'. First the whole company is given written exam which relates heavily to operational area SOPS and enemy related info (of which they have hitherto received no training or information). Statistics are produced.

    At the end of the week of 'training' (carried out by 'instructors' in their second year of service with little or no operational service) they are retested and guess what?

    The improvement is dramatic ... (surprise suprise) ... something like a 30% improvement in the standard! Oh what a wonderful training team they are.

    Sound familiar?

  6. #6
    Council Member Ken White's Avatar
    Join Date
    May 2007
    Location
    Florida
    Posts
    8,060

    Default Yes...

    Quote Originally Posted by JMA View Post
    I have specific experience of this insanity.
    . . .
    Sound familiar?
    Too familiar.

    Same sort of thing. US Army ROTC Advanced Camp at Bragg, the DCG of then Third Army decided all Cadets should qualify as Sharpshooter or better on the Range. Only available time for refires for those who did not do so was a Sunday. Voila, All Cadets fired Sharpshooter or better on their first try. Happy General, happy NCOs who worked the ranges, happy Cadets *...

    Viet Nam. Diktat was promulgated that no body count could be claimed lacking a weapon . Predictably, body count went down, weapons collected count went up. Queries from 'Higher Hq' " You're falling behind. What's the problem down there?" Counts reversed. Amazing...

    * The General later wrote an article on leadership in which he cited that incident as an example -- with no trace of irony -- of the 'fact' that you could have perfection if you simply demanded it and obtained metrics to validate performance.

    Duty, HONOR, country. Yes...

  7. #7
    Council Member
    Join Date
    Oct 2005
    Posts
    3,169

    Default

    * The General later wrote an article on leadership in which he cited that incident as an example -- with no trace of irony -- of the 'fact' that you could have perfection if you simply demanded it and obtained metrics to validate performance.
    That about sums it up.

  8. #8
    Council Member Firn's Avatar
    Join Date
    Sep 2009
    Posts
    1,297

    Default

    There once was a business owner who was interviewing people for a division manager position. He decided to hire person who could answer the question "How much is 2+2?"

    The engineer pulled out his slide rule and shuffled it back and forth, and finally announced, "It lies between 3.98 and 4.02."

    The mathematician said, "In two hours I can demonstrate it equals 4 with the following short proof."

    The social worker said, "I don't know the answer, but I a glad that we discussed this important question.

    The attorney stated, "In the case of Svenson vs. the State, 2+2 was declared to be 4."

    The trader asked, "Are you buying or selling?"

    The accountant looked at the business owner for a moment. He got out of his chair, went to see if anyone was listening at the door, then went to the window and pulled the drapes closed.

    Returning to the business owner, the accountant slowly leaned across the desk and said in a low voice:

    "What would you like it to be?"
    In business metrics are very important but they should always be seen as help to understand the means and to reach the goals, not as the mean and the goal itself bar the very basic ones as ROI, ROCE etc. Accounting & Controlling are surprisingly interesting fields with often most revealing and shocking findings. I remember how a board of a European company in which I wanted to invest quite some money offered an interest free 1 million € loan to it's members backed by the value of the convertible bond for which that offer was reserved. Getting the whole bond on less then 30 cents on the € was quite a business with that loaned money, considering the bonds shot up to 110 cents a year later and hover still above the 100 cents. Nice but of course like with options that loan was not listed among the expenses... Shareholder money is clearly interest-free and dilution is an imagined problem


    In the military training the proper use of metrics should be much more difficult to implement then in business. The big problem of a heavy reliance are the big incentives to game the system, training for the score and 'making it work' to earn the benefits.

  9. #9
    Banned
    Join Date
    Mar 2010
    Location
    Durban, South Africa
    Posts
    3,902

    Default

    Quote Originally Posted by Firn View Post
    In business metrics are very important but they should always be seen as help to understand the means and to reach the goals, not as the mean and the goal itself bar the very basic ones as ROI, ROCE etc. Accounting & Controlling are surprisingly interesting fields with often most revealing and shocking findings. I remember how a board of a European company in which I wanted to invest quite some money offered an interest free 1 million € loan to it's members backed by the value of the convertible bond for which that offer was reserved. Getting the whole bond on less then 30 cents on the € was quite a business with that loaned money, considering the bonds shot up to 110 cents a year later and hover still above the 100 cents. Nice but of course like with options that loan was not listed among the expenses... Shareholder money is clearly interest-free and dilution is an imagined problem


    In the military training the proper use of metrics should be much more difficult to implement then in business. The big problem of a heavy reliance are the big incentives to game the system, training for the score and 'making it work' to earn the benefits.
    Business? Its a different world. One which needs to be tolerated and endured rather than savoured. Sydney Jary in the magnificent book '18 Platoon' explains the difference and why good military leaders have a problem adjusting to the 'false' world of business.

    Let us now leave the service ethos, where leadership is appreciated and expected of those in authority. I went into business where I found an almost total absence of leadership qualities. There were only two exceptions, one a retired Brigadier REME, the other a Swiss machine tool manufacturer.

    To succeed in business other characteristics - I will not dignify by calling them qualities - are required. To succeed you require energy, ambition, enthusiasm, plausibility and a degree of intolerant ruthlessness. A cocktail of egocentric characteristics which hardly inspires the trust of subordinates. It does, however, gain the confidence of clients. It sells.
    As a final note on this... had I known when I was a young officer that I was fighting for (and soldiers were dying for) the protection of the assets and profits of the ethically and morally deprived we call 'the captains of industry' I would not have done it.
    Last edited by JMA; 02-04-2012 at 05:18 PM.

Similar Threads

  1. The Perils of Arbitrary and False Precision
    By Jedburgh in forum Intelligence
    Replies: 19
    Last Post: 02-06-2013, 12:10 AM
  2. Fixing Metrics
    By Steve the Planner in forum US Policy, Interest, and Endgame
    Replies: 0
    Last Post: 01-11-2010, 12:05 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •