What we really need is a better crystal ball

[Entropy]

Bill,

I'm skeptical of computational approaches, though further research is warranted.

[Bill Moore]

Posted by davidbfpo,

Having immersed myself in data-mining, data collection and similar subjects I remain a critic of this approach - both in the domestic and overseas expeditionary contexts.

As you should, to date most efforts have produced little of value, but on the other hand (and to my knowledge they are not published publicly) there have been examples of successes that show what is possible now, and of course more will be possible in the future.

Of course a lot data isn't digitized, and even if it is it isn't readily available so it be fused with other sets of data, or it isn't structured, and the list of challenges goes on and on, but over time many of those challenges can be addressed.

Artificial intelligence will never replace the power of the human brain (I might be an exception) in our life times, but if you look at a cyborg capability where it simply augments our ability to see patterns and potentially identify links between events and actors not previously visible then I think there is considerable value added.

Additionally, and I haven't seen this discussed yet in any of my readings, to maximize these capabilities we will have to re-engineer our information/intelligence collection processes to stream line the process from collection to transition to structured data that is available to be fused by those with access with these programs. Note that this study used open source data from the media, imagine if they were able to also fuse open source with classified data.

We're definitely not there yet, but I think we're on the road to a "better" crystal ball that will only serve to augment the most important element in the process, which is the human mind. The danger is we'll get some people in the system (seen it before) that wants a system that will replace humans, so hopefully we can keep those idiots marginalized.

[Dayuhan]

The danger is we'll get some people in the system (seen it before) that wants a system that will replace humans, so hopefully we can keep those idiots marginalized.

A more immediate danger is that people will manipulate the inputs and the process in order to produce output that suits their biases and agendas.

[Entropy]

IMO, where this approach could be valuable is as a starting point for further research. The computer could spit out something that essentially says, "hey this looks like a departure from the trend, maybe you should take a look at it."

[davidbfpo]

Dug up by another website a 2009 Charlie Rose interview of the CEO of Palantir Technologies:http://www.charlierose.com/view/interview/10549

What was memorable was our systems sit on top of your's (data) and work.

I know from an exchange sometime ago Palantir were regarded as a useful bit of kit, if expensive.

[davidbfpo]

I've twice listened in person to Jeff Jonas, now with IBM, who has to say the least an interesting career path, starting in Las Vegas countering insider and external threats. He has also contributed in the national security field, he is referred to in the context of John Poindexter's Total Information Awareness (TIA) and his 9/11 PPT is amazing - which is attached.

His blog is:www.jeffjonas.typepad.com

Slightly off topic is his emphasis on privacy and liberty can be enhanced in this post-9/11 operating environment.

[Steve the Planner]

Entropy:

I think we are back to an old conversation.

Tracking raw numbers like population changes, migration pressures, poverty indexes, political/administrative boundary disputes, and political instability provides great ballpark markers for "hot spots."

Hot spots should trigger further analysis and monitoring both for tipping points and triggers (often non quantitative like the vegetable vendor setting himself afire in Algeria), and tracking (How are the new regimes in Libya, Egypt aligning with the "will of the people" driving changes.

But all of this presumes we have a robust and systematic analytical core for essentially Worldwide Monitoring (which we do not), and that that core has effective participation, resources and profile in the feedback loops.

Potentials for instability do not always trigger actual threats, so who decides how to target limited resources?

Personally, I monitor the limited news scrawls of economic, trade and business news in Iraq for indications of whether actual stability is returning to the population as a whole versus the political theatre which, in part, has many old actors playing out old themes.

IMHO, public political instability is often not a threat so much as an exercise in threat diffusion (surfacing of grievances).

Stability, in many ways and areas, can be monitored through the nature and content of vehicle flows as a proxy for broad trade patterns, underlying cross-regional and cross-national linkages, and population success (prosperity, willingness to overcome obstacles).

Stability, in many ways and places, is driven by the integrity of land tenure for a permanent population (and thus economic rights and commitments to an area), for which population tracking is key.

To my knowledge, this stuff is just to esoteric, and not being done.

[slapout9]

Link to a Real News Network Interview. The subject of the interview is not directly related to this thread but if you listen to the interview you will hear how something like the Jonas system The (that David posted a link to) was used to determine this information.



http://www.youtube.com/watch?v=kEg4X...&feature=feedu

[davidbfpo]

Hat tip to the Australian Lowry Institute think-tank for this. A mix of experts commenting on the issue, on-line summary or a podcast:http://www.freakonomics.com/2011/09/...of-prediction/

[slapout9]

Hat tip to the Australian Lowry Institute think-tank for this. A mix of experts commenting on the issue, on-line summary or a podcast:http://www.freakonomics.com/2011/09/...of-prediction/

Here is my question. Isn't the act of saying you cain't predict the future....an act of prediction?

[Steve the Planner]

Slap:

I have one of those T-shirts that says:

Not everyone wandering around is lost.

[davidbfpo]

Difficult to find a place for this commentary on intelligence analysis, so dropped in here.

Titled 'How Critics of Obama's Libya Response Profoundly Misunderstand Intelligence' and sub-titled 'Agencies still don't have all the facts about what went down in Benghazi, and interpreting them correctly will take time, a former CIA analyst explains':http://www.theatlantic.com/politics/...igence/263139/

When intelligence from a conflict zone is assessed, the results are not clear, linear, or static. Rather, 21st-century intelligence analysis -- particularly when it is occurring in real-time and on something high-profile -- can be messy, obtuse and, above all, evolving.

1. A lot of first-contact intelligence is wrong.
2. Intelligence analysts almost always hedge their language.
3. The intelligence community's production timelines are ill-suited to our 24/7 news cycle.
4. Arguing over what to call the assailants misses the point.

The author co-wrote this book, not heard of:http://www.amazon.com/Find-Fix-Finis.../dp/1610391284

[Ken White]

I agree with his or her contention that political gamesmanship and a rush to blame any sitting President -- Obama is far from the first to suffer that -- is ill advised and just wrong. I disagree rather strongly on the contentions as to why that is so:

1. A lot of first-contact intelligence is wrong.

That is true, however, my observation has been that most of it is reasonably accurate and much of that is rejected by the 'analysts' as it it does not fit their preconceived notions of what should be. The unnamned author even gives an example of that often fatal failing:

Here's an example: Ten years ago this month, D.C.-area residents were held hostage by the rampages of the Beltway sniper. Over the course of three weeks, the killer slaughtered 10 people and injured others, mostly at random. Based on reasonable FBI and local law-enforcement analysis, the killer was said to be a lone, white, employed, male gunman in a white van. It took additional information and some dumb luck to determine that, other than his gender, every one of these assumptions was totally wrong.

Analysts too often reject facts due to 'assessments' that such facts should not be true. The 9/11 assaults are another example as was the Battle of the Bulge and more than one other military debacle...

His other points also err IMO:

2. Intelligence analysts almost always hedge their language.

He or she is ferociously understating that problem. History is rife with examples of such hedging due to a fear of being wrong. ' Reputations' must be protected, if others die due to that, tough...

3. The intelligence community's production timelines are ill-suited to our 24/7 news cycle.

Not really. The intelligence communities bureaucratic protective instincts are ill suited to that cycle. Not the same thing at all.

4. Arguing over what to call the assailants misses the point.

Well, the author got one out of four correct...

However, in so doing, he or she neatly obscures an issue -- motive. What the assailant is called is indeed immaterial; who the assailant was and what their motivation happened to be are often crucial. The latter will frequently lead to the former.

[davidbfpo]

In April 2012 a London-based left of centre think tank, Demos, published a report; which I read and forgot to post here The three authors include Sir David Omand, one of Whitehall's respected intelligence guru's; which made it more interesting to read.

Link:http://www.demos.co.uk/publications/intelligence

The growth of social media poses a dilemma for security and law enforcement agencies. On the one hand, social media could provide a new form of intelligence – SOCMINT – that could contribute decisively to keeping the public safe. On the other, national security is dependent on public understanding and support for the measures being taken to keep us safe.
Social media challenges current conceptions about privacy, consent and personal data, and new forms of technology allow for more invisible and widespread intrusive surveillance than ever before. Furthermore, analysis of social media for intelligence purposes does not fit easily into the policy and legal frameworks that guarantee that such activity is proportionate, necessary and accountable.

This paper is the first effort to examine the ethical, legal and operational challenges involved in using social media for intelligence and insight purposes.

A "lurker" who works in this field commented:

a thoughtful analysis...they avoid that can of worms as they are keen to discuss the ethical / legal framework that would be needed to support this

The Frontline Club, London held a discussion evening after the launch, rightly the title was 'Cyber-snooping a threat to freedom or a necessary safeguard' and is available on a podcast:http://www.frontlineclub.com/events/...safeguard.html

[davidbfpo]

A "lurker" commended I look at this previously unknown blogsite for a review of the Demos paper and a related UN paper:http://osintblog.org/?p=1462

Good points made here, which to date are rarely heard in public discussions with officialdom on social media intelligence:

A democratic state will not want (and be able) to afford general and permanent mistrust in its ways of safeguarding democracy, and so the authors adjust some Just War criteria in order to form the very necessary ‘rules of engagement’ for SOCMINT: sufficient, sustainable cause; integrity of motive; proportionate and necessary methods; and right authority, validated by external oversight.