The Perils of Arbitrary and False Precision

[Van]

The confusion between accuracy and precision is one of my personal hot-buttons. As an example of the problem...

A guy is measuring things (doesn't really matter what) and only has tools to measure to a tenth of an increment (1.2). He starts talking about 80% of this quantity, and insists that it is exactly .96 (1.2 x .8), without realizing that this is stupid, as he can't measure to this degree of precision, where describing it as "between .9 and 1.0" makes sense and is accurate.

Another example;
A guy with a watch that is only marked 1 through 12 without hashmarks for minutes can be on time every time for appointments because his watch is accurate, gains or loses only a couple of seconds a day, and reflects the time on Naval Observatory's atomic clock, but without precision. The guy with the display down to seconds is always late because his watch loses two minutes a day, or possess greater precision by two orders of magnitude without being accurate.

But words like "about" and ending in "-ish" disturb the harmony of the Type A personalities that call the shots, so they demand a precise and wrong number rather than a less precise but accurate number.

And this is why I argued that CAS3 should include a block on statistics (and got told to sit down and shut up).

[120mm]

But words like "about" and ending in "-ish" disturb the harmony of the Type A personalities that call the shots, so they demand a precise and wrong number rather than a less precise but accurate number.

Especially the mouth-breaking jerkoffs who try to make you come up with a binding number on the spot, in a scene reminiscent of "A Few Good Men".

Dude, I just told you my timeline was completely dependent on a house of cards. How the heck do you expect me to make a time estimate that you will tie me to, later....

[Ron Humphrey]

And this is why I argued that CAS3 should include a block on statistics (and got told to sit down and shut up).

My Stats course did do a lot for me in terms of remembering to take metrics with the BIIIIIG bucket of salt they require

And I really liked your analogy considering that although the extreme precision watch is problematic it doesnt change the fact that the regular watch is still a ton better than trying to guess what time it is by looking to the sky(especially at night)

Metric use -Good
Metric abuse -bad
Metric Dependancy- Requires a lot of therapy and tons of CS(Common Sense) checks in order to be addressed

[selil]

In my area we deal with accuracy versus reliability especially with assessments. Lots of pictures of arrows on targets which I really understand.

[Ron Humphrey]

In my area we deal with accuracy versus reliability especially with assessments. Lots of pictures of arrows on targets which I really understand.

I think the biggest thing that bothers me is how useful metrics can be used correctly and in context with the purpose for which they are being used.

And how more often than not this does not end up being the case due to a variety of things not the least of which is ignorance of what some are actually trying to use them for.

Accuracy has always seemed rather interesting to me in that we actually tend to believe we are creating an accurate picture or understanding of something yet when it all comes down to ground truth we are suddenly amazed by how little our "accurate" assesments actually got right.

I propose that if whatever one is working with or through is not both
(Reliably Accurate) than the greatest part of what we end up with will continue to be right only about 50 percent of the time.


This is one reason I am a strong supporter of trendal analysis because although it uses most of the same metrics you know before you even start that what you end up will not be answers but rather guidelines/possibilities.

So you always get what you expected No more, No Less

[selil]

When I am assessing the amount of information that students have ended up with in a class (based on all of the learning methods utilized and all experiences in and out of class) and based on the learning literacy of the assessment I can find nice gaussian bell curves. Some students will be bulls eyes with accuracy and reliability. Some students will be reliably wrong. Some will hit all over the target. If students who are otherwise reliable and accurate get something wrong I look to see if it was graded wrong or I taught it wrong. That is how I use accuracy and reliability.

Unfortunately the bell curve has limited utility in much of my research. I deal with binary data that the outliers are the important element. Averages have little in relationship to the rest of the environment. Myself I don't believe very much in trend analysis or other predictive methods. Only that which can be observed. Sure we all do it and it is fun, but prediction even with high reliability is rarely scientific.

[Ron Humphrey]

When I am assessing the amount of information that students have ended up with in a class (based on all of the learning methods utilized and all experiences in and out of class) and based on the learning literacy of the assessment I can find nice gaussian bell curves. Some students will be bulls eyes with accuracy and reliability. Some students will be reliably wrong. Some will hit all over the target. If students who are otherwise reliable and accurate get something wrong I look to see if it was graded wrong or I taught it wrong. That is how I use accuracy and reliability.

Unfortunately the bell curve has limited utility in much of my research. I deal with binary data that the outliers are the important element. Averages have little in relationship to the rest of the environment. Myself I don't believe very much in trend analysis or other predictive methods. Only that which can be observed. Sure we all do it and it is fun, but prediction even with high reliability is rarely scientific.

In relation to Predictive methods but possibly in a different context than you might expect. At its base would it not be reasonable to suggest that that which has often been considered predictive would actually be more acurately referred to as recognitive. By this I mean it seeks to look for similar characteristics to that which it has seen before and simply infer within acceptable bounds to attempt to approach an end solution through that lense.

I think about computer viruses and how although many may differ there are always similarities which if taken as a whole can eventually help to define the actual virus itself and even possibly from whence it came. Same with DOS attacks although they may come in different forms the ability to recognize and react to them allows for an almost predictive quality to ones preparations for such attacks.

Or how about finance how many types of applications exist which can at least in some format provide "good enough" answers to provide international level entities to make decisions on how to press forward and stay away from given actions.

Long and Short
I'm not quite sure there's really so much wrong with reasonable predictions based on known historical factors, rather that those predictions should never be blindly followed with upfront expectations that you don't know what you don't know until you get there

[selil]

Long and Short
I'm not quite sure there's really so much wrong with reasonable predictions based on known historical factors, rather that those predictions should never be blindly followed with upfront expectations that you don't know what you don't know until you get there

excellent.