Quote Originally Posted by AnalyticType View Post
You beat me to it!

That's precisely the software I used. I have another one called DecisionWarning which has more graphing tools once you've completed the matrix, but ACH from PARC is better explained and easier to use.

JTF, to answer your third question, the ACH software from PARC (Richards Heuer) has a very detailed tutorial which can be utilized on-screen or downloaded and printed out. As I recall, it prints out to about 50 pages, give or take.

Regarding the subjective elements (credibility and relevance), the options are Low, Medium and High. For credibility of the information, if I had a piece of information that was obtained from a highly credible source, ie the State Department website, OR if I confirmed more than two disparate but credible sources, I rated it High. I used Medium for data which I found in two locations, where one or both sources were less than sterling. And Low I used on a couple items, but then found confirming information elsewhere and changed them to Medium.

Regarding relevance, the same options are available. I was evaluating two different sets of competing hypotheses which were related to each other but not necessarily 'linked' to each other. So I used High relevance for information which strongly was applicable to both sets of hypotheses, and Medium relevance for information which may be peripheral for one set but cogent for the other.

And yes, it is subjective. While I know a fair amount about Latin American history, culture and politics, I have not been there and am not an area expert. Someone who spent a great deal of time there (via State, DoD or CIA, etc) likely would rate relevance or credibility differently than I, in some instances.
This methodology is similar to the Delphi Method. One way to approach the rating is to have a small panel of independent reviewers, then "average" their evaluations. Depending on how you want to look at it, you're either averaging the subjectivity, or averaging it out. I've used this approach in areas such as formal risk assessments for R&D programs with pretty good results.

AT, I owe you a beer if we ever run into each other. I've been looking for a package to do this for some time now.