What we really need is a better crystal ball

[Bill Moore]

http://www.bbc.co.uk/news/technology-14841018

A study, based on millions of articles, charted deteriorating national sentiment ahead of the recent revolutions in Libya and Egypt.

While the analysis was carried out retrospectively, scientists say the same processes could be used to anticipate upcoming conflict.

The system also picked up early clues about Osama Bin Laden's location

Maybe this has some potential, a lot more at:

http://www.uic.edu/htbin/cgiwrap/bin...view/3663/3040

News is increasingly being produced and consumed online, supplanting print and broadcast to represent nearly half of the news monitored across the world today by Western intelligence agencies. Recent literature has suggested that computational analysis of large text archives can yield novel insights to the functioning of society, including predicting future economic events. Applying tone and geographic analysis to a 30–year worldwide news archive, global news tone is found to have forecasted the revolutions in Tunisia, Egypt, and Libya, including the removal of Egyptian President Mubarak, predicted the stability of Saudi Arabia (at least through May 2011), estimated Osama Bin Laden’s likely hiding place as a 200–kilometer radius in Northern Pakistan that includes Abbotabad, and offered a new look at the world’s cultural affiliations. Along the way, common assertions about the news, such as “news is becoming more negative” and “American news portrays a U.S.–centric view of the world” are found to have merit.

[davidbfpo]

I'd forgotten this thread.

Having immersed myself in data-mining, data collection and similar subjects I remain a critic of this approach - both in the domestic and overseas expeditionary contexts.

Sticking to the 'Small Wars' context and the strong possibility they will happen in the developing world - where will the data come from? Even allowing for the spread of mobile phones and computers there are large areas of the world which are electronic / data deserts, albeit maybe ones with a sparse population.

Jobu asked this question a few months ago in a RFI:http://council.smallwarsjournal.com/...ad.php?t=13252.

Data collection is fraught with problems, so is access to such places and how much expertise exists outside?

A 'crystal ball' with very little data and insight is what we have now and for sometime to come.

If our enemies reduce their electronic exposure and do not travel widely the IT solutions so beloved of late will simply not work.

[Entropy]

Bill,

I'm skeptical of computational approaches, though further research is warranted.

[Bill Moore]

Posted by davidbfpo,

Having immersed myself in data-mining, data collection and similar subjects I remain a critic of this approach - both in the domestic and overseas expeditionary contexts.

As you should, to date most efforts have produced little of value, but on the other hand (and to my knowledge they are not published publicly) there have been examples of successes that show what is possible now, and of course more will be possible in the future.

Of course a lot data isn't digitized, and even if it is it isn't readily available so it be fused with other sets of data, or it isn't structured, and the list of challenges goes on and on, but over time many of those challenges can be addressed.

Artificial intelligence will never replace the power of the human brain (I might be an exception) in our life times, but if you look at a cyborg capability where it simply augments our ability to see patterns and potentially identify links between events and actors not previously visible then I think there is considerable value added.

Additionally, and I haven't seen this discussed yet in any of my readings, to maximize these capabilities we will have to re-engineer our information/intelligence collection processes to stream line the process from collection to transition to structured data that is available to be fused by those with access with these programs. Note that this study used open source data from the media, imagine if they were able to also fuse open source with classified data.

We're definitely not there yet, but I think we're on the road to a "better" crystal ball that will only serve to augment the most important element in the process, which is the human mind. The danger is we'll get some people in the system (seen it before) that wants a system that will replace humans, so hopefully we can keep those idiots marginalized.

[Dayuhan]

The danger is we'll get some people in the system (seen it before) that wants a system that will replace humans, so hopefully we can keep those idiots marginalized.

A more immediate danger is that people will manipulate the inputs and the process in order to produce output that suits their biases and agendas.

[Entropy]

IMO, where this approach could be valuable is as a starting point for further research. The computer could spit out something that essentially says, "hey this looks like a departure from the trend, maybe you should take a look at it."

[davidbfpo]

Dug up by another website a 2009 Charlie Rose interview of the CEO of Palantir Technologies:http://www.charlierose.com/view/interview/10549

What was memorable was our systems sit on top of your's (data) and work.

I know from an exchange sometime ago Palantir were regarded as a useful bit of kit, if expensive.

[davidbfpo]

I've twice listened in person to Jeff Jonas, now with IBM, who has to say the least an interesting career path, starting in Las Vegas countering insider and external threats. He has also contributed in the national security field, he is referred to in the context of John Poindexter's Total Information Awareness (TIA) and his 9/11 PPT is amazing - which is attached.

His blog is:www.jeffjonas.typepad.com

Slightly off topic is his emphasis on privacy and liberty can be enhanced in this post-9/11 operating environment.

[Steve the Planner]

Entropy:

I think we are back to an old conversation.

Tracking raw numbers like population changes, migration pressures, poverty indexes, political/administrative boundary disputes, and political instability provides great ballpark markers for "hot spots."

Hot spots should trigger further analysis and monitoring both for tipping points and triggers (often non quantitative like the vegetable vendor setting himself afire in Algeria), and tracking (How are the new regimes in Libya, Egypt aligning with the "will of the people" driving changes.

But all of this presumes we have a robust and systematic analytical core for essentially Worldwide Monitoring (which we do not), and that that core has effective participation, resources and profile in the feedback loops.

Potentials for instability do not always trigger actual threats, so who decides how to target limited resources?

Personally, I monitor the limited news scrawls of economic, trade and business news in Iraq for indications of whether actual stability is returning to the population as a whole versus the political theatre which, in part, has many old actors playing out old themes.

IMHO, public political instability is often not a threat so much as an exercise in threat diffusion (surfacing of grievances).

Stability, in many ways and areas, can be monitored through the nature and content of vehicle flows as a proxy for broad trade patterns, underlying cross-regional and cross-national linkages, and population success (prosperity, willingness to overcome obstacles).

Stability, in many ways and places, is driven by the integrity of land tenure for a permanent population (and thus economic rights and commitments to an area), for which population tracking is key.

To my knowledge, this stuff is just to esoteric, and not being done.

[slapout9]

Link to a Real News Network Interview. The subject of the interview is not directly related to this thread but if you listen to the interview you will hear how something like the Jonas system The (that David posted a link to) was used to determine this information.



http://www.youtube.com/watch?v=kEg4X...&feature=feedu