Do Metrics Matter?

[Bill Moore]

Members of SWJ often debate the value of metrics and assessments on other threads, but to my knowledge I don't think we have a thread dedicated to the debate/discussion. I'm starting this one based on a discussion/debate that has emerged on SWJ Journal at this link http://smallwarsjournal.com/jrnl/art...can-way-of-war

There are more posts at the site, but I'll start the discussion with a couple of them here:

agree that VSO has been "sickened by the curse of McNamara"- although I'd expand that to the entire DoD has been sickened by it. Rumsfeld's (and Gates' and Hagels') expansion upon McNamara's philosophy: that one can apply reductive analytics to social phenomena (war being a social phenomenon in my mind) has turned us more systematic, not less. The systems demand metrics- and thus every CBA I'm involved in concludes we need more IT systems platforms to better aggregate the data for us- because if only our data systems gave us more simplification of the complexity out there... we'd be golden...

Even if you cannot measure entire complexity of "complex causal relationship", still there are some aspects which are measurable (and are quite useful). Is is possible to measure percentage of villages where I established VSO program (MoP)? It is. Is it possible to determine whether the program is working or not in those villages (MoE)? Sure (Depends if I chose correct criteria). Is it possible to determine whether the insurgents have access to the populace and to what extent? In one village? In some percent of villages in a district? Countrywise? Sure. Is it possible to measure it geographically? Sure. Is it possible to determine the changes in time and space? Yes.

The real problem is not using those measurements per se but using them honestly and correctly.

My response:

You argue you can't plan without metrics which is absolutely ludicrous, unless you're a businessman. Metrics didn't become a major factor in U.S. military campaign planning until McNamara introduced modern industrial practices into DOD. Metrics often have no value based on the fact our leadership will always skew them to fit their preconceived perceptions. Even if we could assess metrics quantitatively or qualitatively without bias (we can't), the results would still mean little. We're not trying to increase our profit margins, the only thing that ultimately matters in a conflict is the adversary's will and means to continue fighting. As long as this persists the conflict will persist regardless of our overly positive or negative metrics. This is what we have seen to forgotten over the past 50 or so years. This doesn't mean we have defeat the adversary via military force, we could reach a political compromise, quit, etc., but regardless of the means our measuring of random and irrelevant metrics means nothing and distracts us from what is important. In fact, I think it can result in creating a distorted reality where we convince ourselves we're winning because we're focused on nonsense that is irrelevant to defeating or acquiescing to the adversary.

If we're fighting the Taliban in Afghanistan, and admittedly that is debatable, then measuring metrics related to economic development, number of villages allegedly pacified by the VSO program, number of troops trained, or even the number of enemy contacts ultimately means very little if the Taliban is still willing to fight. Of course the conflict is more complex than our fight with the Taliban and most recognize that. If we followed Gant's advice I suspect we would have accelerated reinitiating tribal warfare that existed in Afghanistan prior to the arrival of the Taliban by enhancing the means of each tribe to fight each other, and they would play us like a maestro plays a violin.

More rational comments below.

[Bill Moore]

While my argument is too absolute above, the underlying point in my opinion remains valid which is that we have allowed metrics (the tail) to wag the dog (strategy). Our search for the perfect metric will always be illusionary, but that doesn't mean some limited metrics and assessments don't have a role in conflict, but that role should be much more limited than it is now. Metrics and assessments will never tell us if we're winning if winning means defeating an adversary, or getting the adversary to stop fighting (political agreement, etc.). As I stated above what we measure has little impact on the adversary's will or means to continue fighting for whatever his objectives may be. When metrics distract us from this reality we lose sight of what is really important. I'm sure our metrics would have told us Germany and Japan should have surrendered long before they did based on the futility of continued struggle, but people are proud and stubborn beyond reason. The same holds true for insurgents and the counterinsurgent who both wage war/conflict until both sides tire enough to resolve their differences politically. All our measures/assessments prior to that point are of questionable value. Providing a couple of starting papers to inform the debate that I hope will follow since this is an important topic for all conflict, not just small wars.

https://csis.org/publication/afghani...-ten-years-war

Afghanistan: The Failed Metrics of Ten Years of War

This analysis looks at the reporting available on the state of the war at the end of 2011, in terms of the data, trends, and maps available from the US Department of Defense (DoD), the US National Counterterrorism Center (NCTC), the NATO/ISAF command, and the UN. It attempts to explore the meaning of these data, the reasons for the sharp differences between them, and what they say about the fighting to date and its progress.

http://wih.sagepub.com/content/19/1/...ication_detail

The Problem of Metrics:
Assessing Progress and Effectiveness in the Vietnam War

A lot in this article, but I found this particular paragraph of particular interest.

South East Asia expert Bernard B. Fall similarly highlighted the complexities of measuring success in an unconventional environment. Fall had spent nearly 15 years in Indochina becoming one of its most respected scholars. His 1961 publication Street without Joy, a classic of the eight-year French-Indochina War, influenced a wide range of American officers preparing for deployment to Vietnam. Fall defined victory in revolutionary war as ‘the people and the army … emerg[ing] on the same side of the fight’.12 He realized, though, that assessing progress toward such victory required suitable indicators.
The French criteria for ‘control’ often had ‘no real meaning when it came to giving a factual picture of who owned what (or whom) throughout the Vietnamese countryside inside the French battleline, much less outside’.13 Fall argued that trends in levels of security and population control could be plotted objectively on a map, given accurate reporting of assassinations, insurgent raids, and Vietcong taxation. Measuring ‘administrative control’, however difficult, if done properly, provided military commanders with the most accurate assessment of their progress.14

O.K. I admit my bias I a big Bernard Fall fan, and yes it appears he is making an argument for metrics, but hey a big fan doesn't mean a die hard fan, we can have our disagreements. On a serious note, I recall reading comments Mr. Fall made in an article about how the U.S. village assessment process was highly inaccurate in Vietnam, and while that article didn't use the term administrative control his example did. He pointed to the schools we built (sound familiar, Kodak moments for our Civil Affairs Teams) being used as a metric of success, but the reality was the Viet Cong administered the schools and appointed the teachers. I had a senior Afghan security official tell me the same thing in 2010. I wasn't able to confirm it, but he said Mullahs were teaching in two of the schools we built radicalizing the children and parents didn't want to send their kids there, but were threatened if they didn't. I suspect our assessments and metrics never captured that uncomfortable truth.

By late 1967 the sheer weight of numbers had become crushing. At MACV commander conferences, staff officers deluged Westmoreland with 65 charts during the Measurement of Progress briefing alone. According to the briefing officer, no senior general expressed any interest in one indicator over another.52 All the while, the US mission in South Vietnam tempered any signs of failure with a blizzard of statistics suggesting progress that led only to confusion on the American home front. Both optimists and pessimists easily justified their positions from the mounds of conflicting data. Without
linkages to coherent strategic aims and sound threat assessments, it seemed any balance sheet or prognosis was as good as the next. Even the establishment of an operations research and systems analysis office in MACV (MACEVAL) and an increasing reliance on automation did little to facilitate analysis of the war’s trends. Given the obsession with statistics and measurement, the war’s complexity had simply overwhelmed MACV’s
capacity for understanding.53

and part of the last paragraph captures it all rather nicely

In the end, the United States Army failed in Vietnam in part because its metrics for success masked important operational and organizational deficiencies. Flawed measurements validated imperfect counter-insurgency methods and provided MACV with a false sense of progress and effectiveness. These measurements were symptomatic of a larger failure in thinking about the war’s deeper issues.

[Bob's World]

IMO, the biggest problem is that we insist on forcing every conflict, regardless of nature or form, into the Clausewitzian war box.

I think we are better served if we recognize two broad categories of conflict, each of which requires a completely different understanding and logic to understand, address or assess.

The first of these is your classic interstate conflict, or "state-based conflict". This is war and apply Clausewitz liberally (or literally) as you choose. While having a smart strategy is preferable, one can win through tactical victories regardless of their strategic shortfalls. All one must do is sieze and hold ground, defeat or destroy warmaking capacity, and coerce the Army, Government (and people if one intends to stay) of the opponent and one wins. This lends itself to neat metrics of tanks destoyed, miles advanced, rivers crossed, etc, etc, etc. In short, the sum of tactics will ultimately add up to strategy. Objective measures are relevant. Each commander can measure what he did in his sector during his tenure to assess his personal and unit success. These separate reports can then be added to tell a comprehensive story of if one is "winning" or "losing."

The other category of conflict is intrastate conflict, or "populace-based conflict." This may be war, but often is not war at all and is better thought of as an extreme form of civil emergency. In these types of conflicts success does not come from defeating the capacity of the challenger's ability to fight or controlling his terrain, or capturing his flag. Rather, success comes from understanding and repairing the perceptions of governance resident among some population (or populations) that have been driven (or neglected) to the point where they feel compelled to illegally, and often violently, challenge these existing, offending, systems of governance.

In these types of conflicts all tactical action should be designed and executed with primary strategic purpose of shaping these perceptions in positive directions, even when the immediate, tactical purpose my be to defeat some force, clear some village, or extend electricity to a region previously dark. One can certainly measure in objective ways all of those tactical actions - but all one learns from such measures is what one has done. Much more important to appreciate is HOW the actions were done, and what effect they had on the critical perceptions of the populations the illegal challengers are coming from - but also non-problematic domestic populations and populations abroad with only indirect or perhaps no stake at all in the conflict.

This is incredibly subjective. It cannot be measured during a single commander's tenure, nor within the confines of his battle space. One cannot tell which commander is good or bad, as what they did that can be measured is largely meaningless in of itself - rather it is how they did what they did and how those actions were perceive that matters. That, however only manifests over time and is impossible to attribute to any particular man, unit or action.

We must evolve in how we think about conflict in general before we can evolve in how we assess the effectiveness of our actions. I personally see little inclination to learn or evolve in that direction. Meanwhile we have our "Made in Iraq" generals (I see HR McMaster is nominated for his 3rd star) and our "Made in Afghanistan" generals. All possible because we could objectively measure the superiority of these commanders over their peers by what they did during their tenures and within their battle space. We celebrate tactical excellence, and turn a blind eye to strategic failure.

Strategic failure in the face of such tactical excellence cannot be our fault, so it is blamed on convenient foils, such as: "complexity" or "ideology" or host governments/security forces/populations (or our own at home) who "lack the will" to win. We widely recognize that the global strategic environment has fundamentlaly changed and that power is shifting to be more balanced between those within government and those outside of government. Yet we cling to our old thinking and practices.


Bottom line is that we suck stategically at populace-based conflicts primarily because it is we who lack the skill or will to recognize that they are fundamentally different than state-based conflicts. Afterall, doing so would require a much fuller accounting of the causal role of governments (the local one as well as those who intervene) in these conflicts - and besides, how would we know who to promote??

[slapout9]

You are dealing with Systems and not countries or in many cases not even governments as we think of them. Until we accept this and begin to learn about reality as it is as opposed to viewing it through the eyes of dead people...we are just doomed. Link to video on System of Systems.

http://www.youtube.com/watch?v=h2br2_twHfw

[Steve Blair]

Metrics can be useful, but the issue I keep seeing in many areas is that they often become an end in themselves. And that's when they become dangerous. In law enforcement you see crime reporting twisted to keep certain "hot" categories low, in academia you see actual student learning subordinated to metrics based on five year completion rates and situations where resources are focused on the top 5% of students and the bottom 5%, leaving the average student in the morass of being "average" and thus not requiring resources.

A metric is a tool. Nothing more. Elevating it to a goal is dangerous, especially when you're using the wrong metric or trying to quantify something that can't easily be quantified (or shouldn't be quantified).

[davidbfpo]

I have long argued - in the UK policing context - that 'Key Performance Indicators' were twisted into 'Measures' or metrics. A very different function.

Rudy Giuliani in his book on NYC writes well on the need for having metrics to assess performance, IIRC he called them performance measures.