The Perils of Arbitrary and False Precision

[Jedburgh]

Kent's Imperative, 22 Jan 08: The Perils of Arbitrary and False Precision

We find quite unhelpful the recent academic obsessions over estimative language – largely an exercise in the introduction of a numerical system which offers a degree of apparently comforting but entirely arbitrary, and therefore utterly false, precision. It seems however that we are in one of those cycles which seem to come along in the intelligence community every few decades or so, in which the numerologists and other soothsayers attempt to reshape the profession into their own desires for a more “scientific” practice.

Let us be clear. There are times when quantitative analytic methodology is vital – but there are far more situations in which it is misapplied, misunderstood, and entirely out of place. The latter comprise the vast majority of scenarios in which analytic tradecraft is called upon – not the least of which may be attributed to the highly unbounded and indeterminate nature of the problems with which we must grapple. And any time in which a quantitative basis has not been established, the insertion of numerical percentages for predictive purposes is little more than a farcical exercise in arbitrary selection......

[J Wolfsberger]

Originally Posted by Jedburgh

Kent's Imperative, 22 Jan 08: The Perils of Arbitrary and False Precision

This has been a long time problem in system analysis, engineering and supporting modeling/simulation. Accuracy without precision can still be useful. Precision without accuracy is not only useless, it is almost always outright damaging since it leads people to infer a level of accuracy that's absent. In fact, and without citing examples, precision without accuracy is usually nothing more than a sophisticated method for lying.

[Jedburgh]

USIP, 21 Mar 08: Using Quantitative and Qualitative Models to Forecast Instability

Summary

• Preventing violent conflict requires early warning of likely crises so that preventive actions can be planned and taken before the onset of mass violence.

• For most of the post–World War II period, policymakers and intelligence agencies have relied on experts to make qualitative judgments regarding the risk of instability or violent changes in their areas of study. Yet the inability of such experts to adequately predict major events has led to efforts to use social and analytical tools to create more “scientific” forecasts of political crises.

• The advent of quantitative forecasting models that give early warning of the onset of political instability offers the prospect of major advances in the accuracy of forecasting over traditional qualitative methods.

• Because certain models have a demonstrated accuracy of over 80 percent in early identification of political crises, some have questioned whether such models should replace traditional qualitative analysis.

• While these quantitative forecasting methods should move to the foreground and play a key role in developing early warning tools, this does not mean that traditional qualitative analysis is dispensable.

• The best results for early warning are most likely obtained by the judicious combination of quantitative analysis based on forecasting models with qualitative analysis that rests on explicit causal relationships and precise forecasts of its own.

• Policymakers and analysts should insist on a multiple-method approach, which has greater forecasting power than either the quantitative or qualitative method alone. In this way, political instability forecasting is likely to make its largest advance over earlier practices.

Complete 16 page paper at the link.

[120mm]

A fine example of false precision, and using quantitative terms where they do not apply is in the following "study":

http://www.rd.com/national-interest/...s/article.html

A recent Reader's Digest article, where they rank colleges for safety, using 19 different variables.

The problem with the study? Iowa State University of Ames, Iowa is rated as the second most unsafe campus in the US. (my alma mater, obtw) Anyone who has ever been to Ames, Iowa, will admit that something, somewhere has to be screwed up if ISU is quantified as the second least safe campus in America.

Or that the University of Worcester is like 4 from the bottom, while Boston U. is 4 from the top in safety ratings to find an "apples to apples" comparison. Anyone with a full and functional brain pan could walk around either campus and realize that Boston U. (my "other" alma mater) is probably a less safe place to be than Worcester.

Frankly, they chose largely irrelevant variables vis-a-vis safety, and then assigned them arbitrary values, which they then took very seriously in their analysis.

In other words, most of these folks are full of crap.

[Hacksaw]

Clearly we have always been inflicted with those who are apt to assign numeric values to everything from individual military performance to Corps COA Evaluation. Specifically why most people like this type of analysis is probably beyond knowing, but I offer the following as possibilities...

1. It is comforting for some (even if they know it is intellectually dishonest) to be able to point to a numeric outcome as a rationale for their decisions. [I]I think this is because they sense it provides a level of freedom from culpability, "but the numbers said this was the best COA, I shall have the offending staff officer shot!"

2. It provides a level of uniformity across the force, "If we all use the same criteria (and associated definitions), we will all come to the same conclusion." I call any reader's attention to the big CAS3 decision brief exercise that a generation of officers were forced to grind out complete with decmat and pairwise comparison.

3. We like to kid ourselves that we can, thru calculations, remove randomness, complexity, and chaos from the nature of our business. Which we all know is horse feathers....

The hardest trick I ever had with Division Commanders (planned for four) was fostering the idea that if we got everyone, metaphorically speaking, marching somewhat in the direction... that was a very good thing. And that sometimes, despite our best efforts, an action taken has unintended/unpredicted consequences. The best we can do is develop plans so that we can account/react/mitigate the effects when those occur.

I batted 50% with those four, one went on to be the hero of New Orleans, the other the hero of Mosul. A coincidence perhaps, but I think crises helps focus the mind.

Live well and row

[Ken White]

Originally Posted by Hacksaw

...
I batted 50% with those four, one went on to be the hero of New Orleans, the other the hero of Mosul. A coincidence perhaps, but I think crises helps focus the mind.

Of Ol' Russ is deserving of accolades.

Or many days resting and loafing in the shade...

[Jedburgh]

Edge, 15 Sep 08: The Fourth Quadrant: A Map of the Limits of Statistics

......In the following Edge original essay, Taleb continues his examination of Black Swans, the highly improbable and unpredictable events that have massive impact. He claims that those who are putting society at risk are "no true statisticians", merely people using statistics either without understanding them, or in a self-serving manner. "The current subprime crisis did wonders to help me drill my point about the limits of statistically driven claims," he says.

Taleb, looking at the cataclysmic situation facing financial institutions today, points out that "the banking system, betting against Black Swans, has lost over 1 Trillion dollars (so far), more than was ever made in the history of banking".

But, as he points out, there is also good news.

We can identify where the danger zone is located, which I call "the fourth quadrant", and show it on a map with more or less clear boundaries. A map is a useful thing because you know where you are safe and where your knowledge is questionable. So I drew for the Edge readers a tableau showing the boundaries where statistics works well and where it is questionable or unreliable. Now once you identify where the danger zone is, where your knowledge is no longer valid, you can easily make some policy rules: how to conduct yourself in that fourth quadrant; what to avoid......

[Van]

The confusion between accuracy and precision is one of my personal hot-buttons. As an example of the problem...

A guy is measuring things (doesn't really matter what) and only has tools to measure to a tenth of an increment (1.2). He starts talking about 80% of this quantity, and insists that it is exactly .96 (1.2 x .8), without realizing that this is stupid, as he can't measure to this degree of precision, where describing it as "between .9 and 1.0" makes sense and is accurate.

Another example;
A guy with a watch that is only marked 1 through 12 without hashmarks for minutes can be on time every time for appointments because his watch is accurate, gains or loses only a couple of seconds a day, and reflects the time on Naval Observatory's atomic clock, but without precision. The guy with the display down to seconds is always late because his watch loses two minutes a day, or possess greater precision by two orders of magnitude without being accurate.

But words like "about" and ending in "-ish" disturb the harmony of the Type A personalities that call the shots, so they demand a precise and wrong number rather than a less precise but accurate number.

And this is why I argued that CAS3 should include a block on statistics (and got told to sit down and shut up).

[120mm]

Originally Posted by Van

But words like "about" and ending in "-ish" disturb the harmony of the Type A personalities that call the shots, so they demand a precise and wrong number rather than a less precise but accurate number.

Especially the mouth-breaking jerkoffs who try to make you come up with a binding number on the spot, in a scene reminiscent of "A Few Good Men".

Dude, I just told you my timeline was completely dependent on a house of cards. How the heck do you expect me to make a time estimate that you will tie me to, later....

[Ron Humphrey]

Originally Posted by Van

And this is why I argued that CAS3 should include a block on statistics (and got told to sit down and shut up).

My Stats course did do a lot for me in terms of remembering to take metrics with the BIIIIIG bucket of salt they require

And I really liked your analogy considering that although the extreme precision watch is problematic it doesnt change the fact that the regular watch is still a ton better than trying to guess what time it is by looking to the sky(especially at night)

Metric use -Good
Metric abuse -bad
Metric Dependancy- Requires a lot of therapy and tons of CS(Common Sense) checks in order to be addressed

[selil]

In my area we deal with accuracy versus reliability especially with assessments. Lots of pictures of arrows on targets which I really understand.

[Ron Humphrey]

Originally Posted by selil

In my area we deal with accuracy versus reliability especially with assessments. Lots of pictures of arrows on targets which I really understand.

I think the biggest thing that bothers me is how useful metrics can be used correctly and in context with the purpose for which they are being used.

And how more often than not this does not end up being the case due to a variety of things not the least of which is ignorance of what some are actually trying to use them for.

Accuracy has always seemed rather interesting to me in that we actually tend to believe we are creating an accurate picture or understanding of something yet when it all comes down to ground truth we are suddenly amazed by how little our "accurate" assesments actually got right.

I propose that if whatever one is working with or through is not both
(Reliably Accurate) than the greatest part of what we end up with will continue to be right only about 50 percent of the time.


This is one reason I am a strong supporter of trendal analysis because although it uses most of the same metrics you know before you even start that what you end up will not be answers but rather guidelines/possibilities.

So you always get what you expected No more, No Less

[selil]

When I am assessing the amount of information that students have ended up with in a class (based on all of the learning methods utilized and all experiences in and out of class) and based on the learning literacy of the assessment I can find nice gaussian bell curves. Some students will be bulls eyes with accuracy and reliability. Some students will be reliably wrong. Some will hit all over the target. If students who are otherwise reliable and accurate get something wrong I look to see if it was graded wrong or I taught it wrong. That is how I use accuracy and reliability.

Unfortunately the bell curve has limited utility in much of my research. I deal with binary data that the outliers are the important element. Averages have little in relationship to the rest of the environment. Myself I don't believe very much in trend analysis or other predictive methods. Only that which can be observed. Sure we all do it and it is fun, but prediction even with high reliability is rarely scientific.

[Ron Humphrey]

Originally Posted by selil

When I am assessing the amount of information that students have ended up with in a class (based on all of the learning methods utilized and all experiences in and out of class) and based on the learning literacy of the assessment I can find nice gaussian bell curves. Some students will be bulls eyes with accuracy and reliability. Some students will be reliably wrong. Some will hit all over the target. If students who are otherwise reliable and accurate get something wrong I look to see if it was graded wrong or I taught it wrong. That is how I use accuracy and reliability.

Unfortunately the bell curve has limited utility in much of my research. I deal with binary data that the outliers are the important element. Averages have little in relationship to the rest of the environment. Myself I don't believe very much in trend analysis or other predictive methods. Only that which can be observed. Sure we all do it and it is fun, but prediction even with high reliability is rarely scientific.

In relation to Predictive methods but possibly in a different context than you might expect. At its base would it not be reasonable to suggest that that which has often been considered predictive would actually be more acurately referred to as recognitive. By this I mean it seeks to look for similar characteristics to that which it has seen before and simply infer within acceptable bounds to attempt to approach an end solution through that lense.

I think about computer viruses and how although many may differ there are always similarities which if taken as a whole can eventually help to define the actual virus itself and even possibly from whence it came. Same with DOS attacks although they may come in different forms the ability to recognize and react to them allows for an almost predictive quality to ones preparations for such attacks.

Or how about finance how many types of applications exist which can at least in some format provide "good enough" answers to provide international level entities to make decisions on how to press forward and stay away from given actions.

Long and Short
I'm not quite sure there's really so much wrong with reasonable predictions based on known historical factors, rather that those predictions should never be blindly followed with upfront expectations that you don't know what you don't know until you get there

[selil]

Originally Posted by Ron Humphrey

Long and Short
I'm not quite sure there's really so much wrong with reasonable predictions based on known historical factors, rather that those predictions should never be blindly followed with upfront expectations that you don't know what you don't know until you get there

excellent.

[Van]

Seli, could you define 'reliability' in this context, please?

I hope I am misunderstanding your usage, it sounds like a variation on the accuracy/precision thing.

Also, your gaussian curve of answers makes me antsy. Clustering around different possibilities invites more productive discussion.

[selil]

Originally Posted by Van

Seli, could you define 'reliability' in this context, please?

I hope I am misunderstanding your usage, it sounds like a variation on the accuracy/precision thing.

Also, your gaussian curve of answers makes me antsy. Clustering around different possibilities invites more productive discussion.

It is a variation on the accuracy/precision element.

Reliability/repeatability does not have to be a bulls eye. It can be the same spot on the target but a clear miss. It has an unfortunate tendency to be related to causation but is more indicative of clustering.

The gaussian curve will occur in a large enough set of classes. I have NEVER had a perfect bell with one course of 20 students. I'm just not that consistent and the students, time of day, and so many other variables just keep it from happening. But, feeding five or six courses with an n+100+ and looking at that the curve looks like Monte Carlo data. I do have a course that has a dumbbell shape to them. F's and A's. That is because it has one project and you either pass or you don't. Some don't.

[120mm]

Originally Posted by Van

Seli, could you define 'reliability' in this context, please?

I hope I am misunderstanding your usage, it sounds like a variation on the accuracy/precision thing.

Also, your gaussian curve of answers makes me antsy. Clustering around different possibilities invites more productive discussion.

Van,

I like this website for all my definitional needs:

http://www.socialresearchmethods.net/kb/reliable.php

Drill up or down for a 'splanation of what you need.

[selil]

Originally Posted by 120mm

Van,

I like this website for all my definitional needs:

http://www.socialresearchmethods.net/kb/reliable.php

Drill up or down for a 'splanation of what you need.

And they even have the targets http://www.socialresearchmethods.net/kb/relandval.php

[davidbfpo]

For those who are interested in 'Black Swans' here is French analyst's review:http://www.redanalysis.org/2013/01/2...-of-foresight/

Not that I'm obsessed with Mali; there is a link to a forecast of a coup in Mali in 2012 correctly, after some number crunching:http://dartthrowingchimp.wordpress.c...asts-for-2013/