Open Source Intelligence (OSINT): merged thread

[sgmgrumpy]

Remove if already posted.

On July 16-17, Chris Pallaris, Sean Costigan and I attended the DNI Open Source Intelligence (OSINT) Conference in Washington, DC. The most important message that emerged from the two day session was that OSINT is no longer "nice to have", rather it is an absolute imperative in our contemporary world of complex, asymmetric and non-linear security threats.

Read here descriptions of the panel sessions and download selected power point presentations.

[Dominique R. Poirier]

In my own opinion, a good specialist armed, once and for all, with solid real world knowledge in politics in general and about a given country or region can make surprisingly good analysis, and even accurate forecasts in politics and strategy on the sole basis of open sources.
However, it is understood that, in that case, analysis and forecast are likely to loose in accuracy with time since real world knowledge are not exempt from changes and evolutions. The pace of this observable decrease in accuracy depends of the observed country or region and of the significant political events or upheavals (publicly known or not) happening in them.

I see that correct deductions and forecasts exclusively based upon open sources owe to two ways of analyzing open sources which are: the formal analysis of the open sources per se; and perception management analysis. Other factors linked to the specifics of the observed country (when the object of our attention happens to be a country in particular) will provide the analyst a basis which will help detecting deception attempts. I’ll probably not tell anything new if I say that if the formal content of a given media known for its inclination to practice deception is therefore of relatively poor value, then we may consider it otherwise when analyzing it under the angle of perception management.

What happens when it comes to Army intelligence operations (a field about which I bloody don't know anything)? I admit that things might be ruled otherwise, then; especially if the enemy is not a nation or a country with a government and its own national media and is, or is not, acting as proxy.

So I’ll read this manual with great interest.

[sgmgrumpy]

Open Source Intelligence Familiarization Documents

The intent of this single-page document is to share basic knowledge about Open Source Intelligence (OSINT). From OSS net

Version 2, May 2006

[Jedburgh]

CRS, 5 Dec 07: Open Source Intelligence (OSINT): Issues for Congress

Open source information (OSINT) is derived from newspapers, journals, radio and television, and the Internet. Intelligence analysts have long used such information to supplement classified data, but systematically collecting open source information has not been a priority of the U.S. Intelligence Community (IC). In recent years, given changes in the international environment, there have been calls, from Congress and the 9/11 Commission among others, for a more intense and focused investment in open source collection and analysis. However, some still emphasize that the primary business of intelligence continues to be obtaining and analyzing secrets.

A consensus now exists that OSINT must be systematically collected and should constitute an essential component of analytical products. This has been recognized by various commissions and in statutes. Responding to legislative direction, the Intelligence Community has established the position of Assistant Director of National Intelligence for Open Source and created the National Open Source Center. The goal is to perform specialized OSINT acquisition and analysis functions and create a center of excellence that will support and encourage all intelligence agencies.

The effort has been only underway since late 2005 but the Center is up and running, and providing support, including training, to OSINT professionals throughout the Intelligence Community. Administrative mechanisms are in place to ensure that there is a comprehensive community-wide open source effort. It appears, however, to some observers that not all agencies have as yet made comprehensive commitments to acquiring and using open source information, nor that the ODNI has taken sufficient steps to ensure that open sources are appropriately exploited. Observers suggest that congressional oversight of the OSINT process might provide insight into current progress as well as identify areas that need modification. A particular focus of congressional interest might be potential tradeoffs between classified and open source collection to ensure that needed information is obtained in the best and most cost-effective manner. Proponents maintain that this approach helps to ensure that agents and expensive surveillance systems are focused on obtaining information that is being actively hidden.

The collection and analysis of OSINT information will be ultimately judged by its contribution to the overall intelligence effort. Collecting information from open sources is generally less expensive and less risky than collection from other intelligence sources. The use of OSINT may result not only in monetary savings but also in less risk than utilizing sensitive technical and human sources. OSINT can also provide insights into the types of developments that may not be on the priority list for other systems or may not be susceptible to collection through other intelligence approaches — innovative applications of new technologies, shifts in popular attitudes, emergence of new political and religious movements, growing popular discontent, disillusionment with leadership, etc. Supporters of OSINT maintain that the future contribution of the Intelligence Community will be enhanced by its ability to provide detailed information and incisive analyses of such developments. This report will be updated as new information becomes available.

Complete 27 page report at the link.

[JeffC]

CRS, 5 Dec 07: Open Source Intelligence (OSINT): Issues for Congress

Complete 27 page report at the link.

I just finished reading it during my ferry commute home. I need some time to think about the implications, but I'm concerned about the lack of commitment to OSINT on the part of so many in leadership positions.

I see that DHS is farming out it's OSINT duties to the SITE Institute. I'm inclined to support that model for the entire IC - i.e., farm out OSINT collection and analysis to small niche firms that have the skills, the people, and the agility to do it properly.

[Jedburgh]

I see that DHS is farming out it's OSINT duties to the SITE Institute. I'm inclined to support that model for the entire IC - i.e., farm out OSINT collection and analysis to small niche firms that have the skills, the people, and the agility to do it properly.

Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.

[Rex Brynen]

I completely agree with Ted.

I would also add--as someone who works with both OSINT as well as very high end classified material, that I think its very important that the streams not be too separated, since one is often needed to make sense of the other (certainly for the kind of assessment work I've done).

[JeffC]

Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.

The CRS study that you linked to points out some of the problems that exist within the IC as regards OSINT. Pages 8 and 9 list a few of the major obstacles, but the problems extend beyond this to a lack of confidence in the actual value of OSINT by some senior members of the effected agencies:

As far as why regular analytic production should not be outsourced - no, the reasons for that aren't obvious. In fact, the high percentage of outsourcing at the CIA has been the subject of a lot of debate. What do you think are the reasons for not outsourcing OSINT collection and analysis?

[JeffC]

Remove if already posted.





Read here descriptions of the panel sessions and download selected power point presentations.

Wow. This is a treasure trove. Thanks, grumpy.