Open Source Intelligence (OSINT): merged thread

[sgmgrumpy]

Open Source Intelligence Familiarization Documents

The intent of this single-page document is to share basic knowledge about Open Source Intelligence (OSINT). From OSS net

Version 2, May 2006

[Jedburgh]

CRS, 5 Dec 07: Open Source Intelligence (OSINT): Issues for Congress

Open source information (OSINT) is derived from newspapers, journals, radio and television, and the Internet. Intelligence analysts have long used such information to supplement classified data, but systematically collecting open source information has not been a priority of the U.S. Intelligence Community (IC). In recent years, given changes in the international environment, there have been calls, from Congress and the 9/11 Commission among others, for a more intense and focused investment in open source collection and analysis. However, some still emphasize that the primary business of intelligence continues to be obtaining and analyzing secrets.

A consensus now exists that OSINT must be systematically collected and should constitute an essential component of analytical products. This has been recognized by various commissions and in statutes. Responding to legislative direction, the Intelligence Community has established the position of Assistant Director of National Intelligence for Open Source and created the National Open Source Center. The goal is to perform specialized OSINT acquisition and analysis functions and create a center of excellence that will support and encourage all intelligence agencies.

The effort has been only underway since late 2005 but the Center is up and running, and providing support, including training, to OSINT professionals throughout the Intelligence Community. Administrative mechanisms are in place to ensure that there is a comprehensive community-wide open source effort. It appears, however, to some observers that not all agencies have as yet made comprehensive commitments to acquiring and using open source information, nor that the ODNI has taken sufficient steps to ensure that open sources are appropriately exploited. Observers suggest that congressional oversight of the OSINT process might provide insight into current progress as well as identify areas that need modification. A particular focus of congressional interest might be potential tradeoffs between classified and open source collection to ensure that needed information is obtained in the best and most cost-effective manner. Proponents maintain that this approach helps to ensure that agents and expensive surveillance systems are focused on obtaining information that is being actively hidden.

The collection and analysis of OSINT information will be ultimately judged by its contribution to the overall intelligence effort. Collecting information from open sources is generally less expensive and less risky than collection from other intelligence sources. The use of OSINT may result not only in monetary savings but also in less risk than utilizing sensitive technical and human sources. OSINT can also provide insights into the types of developments that may not be on the priority list for other systems or may not be susceptible to collection through other intelligence approaches — innovative applications of new technologies, shifts in popular attitudes, emergence of new political and religious movements, growing popular discontent, disillusionment with leadership, etc. Supporters of OSINT maintain that the future contribution of the Intelligence Community will be enhanced by its ability to provide detailed information and incisive analyses of such developments. This report will be updated as new information becomes available.

Complete 27 page report at the link.

[JeffC]

CRS, 5 Dec 07: Open Source Intelligence (OSINT): Issues for Congress

Complete 27 page report at the link.

I just finished reading it during my ferry commute home. I need some time to think about the implications, but I'm concerned about the lack of commitment to OSINT on the part of so many in leadership positions.

I see that DHS is farming out it's OSINT duties to the SITE Institute. I'm inclined to support that model for the entire IC - i.e., farm out OSINT collection and analysis to small niche firms that have the skills, the people, and the agility to do it properly.

[Jedburgh]

I see that DHS is farming out it's OSINT duties to the SITE Institute. I'm inclined to support that model for the entire IC - i.e., farm out OSINT collection and analysis to small niche firms that have the skills, the people, and the agility to do it properly.

Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.

[Rex Brynen]

I completely agree with Ted.

I would also add--as someone who works with both OSINT as well as very high end classified material, that I think its very important that the streams not be too separated, since one is often needed to make sense of the other (certainly for the kind of assessment work I've done).

[JeffC]

I completely agree with Ted.

I would also add--as someone who works with both OSINT as well as very high end classified material, that I think its very important that the streams not be too separated, since one is often needed to make sense of the other (certainly for the kind of assessment work I've done).

The CRS report that Ted linked to provides multiple reasons for doubt to be cast on the ability of the IC to fully take advantage of OSINT. The history of Intelligence analysis is a spotted one, so I'm not clear as to why Ted or anyone would think that analysis and collection of OSINT cannot be outsourced for better effect. Since you agree with Ted, perhaps you'll share your reasons?

[Rex Brynen]

The CRS report that Ted linked to provides multiple reasons for doubt to be cast on the ability of the IC to fully take advantage of OSINT. The history of Intelligence analysis is a spotted one, so I'm not clear as to why Ted or anyone would think that analysis and collection of OSINT cannot be outsourced for better effect. Since you agree with Ted, perhaps you'll share your reasons?

I should start by saying the role of OSINT in analysis, the sorts of OSINT we're talking about, the technical challenges of collection and analysis, and its relative weight in production, all vary dramatically depending on the sort of analysis that one is doing. I can really only reflect on its role in the kind of (political and strategic assessment) work that I've done.

Second, it should be noted that there is already a lot of OSINT collection and analysis built into the system on the government side: a good share of diplomatic reporting, for example, is essentially OSINT collection and analysis. That reporting, in turn, feeds into IC product.

My concern about too much outsourcing of OSINT analysis stems from the observation that it is sometimes access to the high end classified material that shapes the relevance of OSINT, and those without access to this aren't always in a good position to understand the significance of what they are seeing in the OSINT material. I must admit, I always like to see data as raw as possible, to determine that the analytical conclusions drawn by others are the same ones that I would also draw.

I'm also worried about a byproduct of too much OSINT outsourcing being the implicit development of a division of labour--"you do the OSINT, we'll do the COMINT/HUMINT/etc"--which runs counter to the fundamental need for analysts to use the full spectrum of material.

I should add that my concerns are much less applicable to the outsourcing of some aspects of collection (which, as noted, is already done), and "preanalysis" of this material of the "hey, this is especially interesting" flag, or the "this relates in interesting ways to previous material" flag.

I have to dash to a conference in 2 minutes, so this answer rambles far more than I would wish. I'm sure, in any event, that Ted will provide a far more thoughtful and coherent answer than I'm capable of!

[Jedburgh]

The quick answer is that contracting out capabilities that should be developed internally weakens rather than strengthens our national intelligence structure.

We've already done long-term damage to military intelligence by "outsourcing" critical capabilities while concurrently reducing internal capabilities for expediency's sake. As we continue to reform and develop our national intelligence capabilities, the last thing we need to do is step out along that path across the IC.

Also, as Rex stated, and I thought was implied in my post, open source information isn't assessed in a vacuum - it is simply one of many sources of information that the analyst must absorb. That is why I said that outsourcing of collection is justified (but only for the short term, until we effectively modify and develop effective internal structures within the IC) but not the outsourcing of analysis.

The "acceptance" of the value of OSINT within the IC is mainly a generational issue. Many "senior" guys are uncomfortable with the broad spectrum of open source media (both currently available and emerging) and prefer working with traditional sources, while the young'uns are perfectly capable of working in both worlds and of effectively integrating the two.

Myself, I find the view of those old boys kind of odd, as OSINT has long played a critical role in strategic intelligence analysis. I'm sure there's a couple of Cold Warriors on the board who recall the effort put into anayzing and interpreting articles published in the official Soviet media to support assessments of powershifts and policy changes in the Kremlin.

[JeffC]

Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.

The CRS study that you linked to points out some of the problems that exist within the IC as regards OSINT. Pages 8 and 9 list a few of the major obstacles, but the problems extend beyond this to a lack of confidence in the actual value of OSINT by some senior members of the effected agencies:

As far as why regular analytic production should not be outsourced - no, the reasons for that aren't obvious. In fact, the high percentage of outsourcing at the CIA has been the subject of a lot of debate. What do you think are the reasons for not outsourcing OSINT collection and analysis?