Open Source Intelligence (OSINT): merged thread

[Jedburgh]

I see that DHS is farming out it's OSINT duties to the SITE Institute. I'm inclined to support that model for the entire IC - i.e., farm out OSINT collection and analysis to small niche firms that have the skills, the people, and the agility to do it properly.

Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.

[Rex Brynen]

I completely agree with Ted.

I would also add--as someone who works with both OSINT as well as very high end classified material, that I think its very important that the streams not be too separated, since one is often needed to make sense of the other (certainly for the kind of assessment work I've done).

[JeffC]

I completely agree with Ted.

I would also add--as someone who works with both OSINT as well as very high end classified material, that I think its very important that the streams not be too separated, since one is often needed to make sense of the other (certainly for the kind of assessment work I've done).

The CRS report that Ted linked to provides multiple reasons for doubt to be cast on the ability of the IC to fully take advantage of OSINT. The history of Intelligence analysis is a spotted one, so I'm not clear as to why Ted or anyone would think that analysis and collection of OSINT cannot be outsourced for better effect. Since you agree with Ted, perhaps you'll share your reasons?

[Rex Brynen]

The CRS report that Ted linked to provides multiple reasons for doubt to be cast on the ability of the IC to fully take advantage of OSINT. The history of Intelligence analysis is a spotted one, so I'm not clear as to why Ted or anyone would think that analysis and collection of OSINT cannot be outsourced for better effect. Since you agree with Ted, perhaps you'll share your reasons?

I should start by saying the role of OSINT in analysis, the sorts of OSINT we're talking about, the technical challenges of collection and analysis, and its relative weight in production, all vary dramatically depending on the sort of analysis that one is doing. I can really only reflect on its role in the kind of (political and strategic assessment) work that I've done.

Second, it should be noted that there is already a lot of OSINT collection and analysis built into the system on the government side: a good share of diplomatic reporting, for example, is essentially OSINT collection and analysis. That reporting, in turn, feeds into IC product.

My concern about too much outsourcing of OSINT analysis stems from the observation that it is sometimes access to the high end classified material that shapes the relevance of OSINT, and those without access to this aren't always in a good position to understand the significance of what they are seeing in the OSINT material. I must admit, I always like to see data as raw as possible, to determine that the analytical conclusions drawn by others are the same ones that I would also draw.

I'm also worried about a byproduct of too much OSINT outsourcing being the implicit development of a division of labour--"you do the OSINT, we'll do the COMINT/HUMINT/etc"--which runs counter to the fundamental need for analysts to use the full spectrum of material.

I should add that my concerns are much less applicable to the outsourcing of some aspects of collection (which, as noted, is already done), and "preanalysis" of this material of the "hey, this is especially interesting" flag, or the "this relates in interesting ways to previous material" flag.

I have to dash to a conference in 2 minutes, so this answer rambles far more than I would wish. I'm sure, in any event, that Ted will provide a far more thoughtful and coherent answer than I'm capable of!

[JeffC]

My concern about too much outsourcing of OSINT analysis stems from the observation that it is sometimes access to the high end classified material that shapes the relevance of OSINT, and those without access to this aren't always in a good position to understand the significance of what they are seeing in the OSINT material. I must admit, I always like to see data as raw as possible, to determine that the analytical conclusions drawn by others are the same ones that I would also draw.

--- In the private sector, this is a given. Evidence is sourced, and anyone who cares to can examine the original source. It's only in the IC that source data is closely guarded rather than shared. It's a good example of one of many flaws in the present system.

I'm also worried about a byproduct of too much OSINT outsourcing being the implicit development of a division of labour--"you do the OSINT, we'll do the COMINT/HUMINT/etc"--which runs counter to the fundamental need for analysts to use the full spectrum of material.

But the current system IS a division of labor. There are MASINT analysts, SIGINT analysts, IMINT analysts, etc., who like the fact that they're specialists and look down their noses at All-Source analysts as nothing more than writers. There's a MITRE study that explores this and recommends some changes be made, but as far as I know, those changes haven't happened yet, and they may never happen at all. Additonally, there are specialist agencies: NRO, NSA, NGA that only focus on certain types of INT collection and analysis.

[Jedburgh]

The quick answer is that contracting out capabilities that should be developed internally weakens rather than strengthens our national intelligence structure.

We've already done long-term damage to military intelligence by "outsourcing" critical capabilities while concurrently reducing internal capabilities for expediency's sake. As we continue to reform and develop our national intelligence capabilities, the last thing we need to do is step out along that path across the IC.

Also, as Rex stated, and I thought was implied in my post, open source information isn't assessed in a vacuum - it is simply one of many sources of information that the analyst must absorb. That is why I said that outsourcing of collection is justified (but only for the short term, until we effectively modify and develop effective internal structures within the IC) but not the outsourcing of analysis.

The "acceptance" of the value of OSINT within the IC is mainly a generational issue. Many "senior" guys are uncomfortable with the broad spectrum of open source media (both currently available and emerging) and prefer working with traditional sources, while the young'uns are perfectly capable of working in both worlds and of effectively integrating the two.

Myself, I find the view of those old boys kind of odd, as OSINT has long played a critical role in strategic intelligence analysis. I'm sure there's a couple of Cold Warriors on the board who recall the effort put into anayzing and interpreting articles published in the official Soviet media to support assessments of powershifts and policy changes in the Kremlin.

[selil]

As has been said the OSINT is but one collection mechanism among many. I take issue with outsourcing it or further diluting the variety information in consideration, but more importantly I take issue with creating another non-permeable barrier in the collection process. If I had to put my finger on the issues with intelligence analysis and processing it is not the volume of information or the speed it is the number of analysts looking at any one area at a time. We need more highly trained analysts inside the intelligence organizations with better tools to do their job better. Outsourcing is not the answer it is only a band-aid over leprosy.

[Penta]

Myself, I find the view of those old boys kind of odd, as OSINT has long played a critical role in strategic intelligence analysis. I'm sure there's a couple of Cold Warriors on the board who recall the effort put into anayzing and interpreting articles published in the official Soviet media to support assessments of powershifts and policy changes in the Kremlin.

To make the elders here feel really old...My advisor at Scranton was (in 61-62) a Soviet analyst for the CIA.

Old-fashioned Kremlinology was often talked about, in a "for those of you who were too young to notice" fashion, in classes. Basically, as history.

Example which sticks out: The immense effort put into getting...photos of the reviewing stands for the May Day and October Revolution parades. Hats or no hats, the positioning of various people, whether they were sitting or standing, analyzed to seemingly-absurd lengths.

How much of that actually produced insights of value, might I ask my elders?

*whistles the Soviet National Anthem boredly*

(I'm young enough to say that...I miss the Cold War. When things were simple, and you knew your enemy was at least sane (and could even be worked with on occasion!). Yes, this points out my youth.)

[JeffC]

The quick answer is that contracting out capabilities that should be developed internally weakens rather than strengthens our national intelligence structure.

We've already done long-term damage to military intelligence by "outsourcing" critical capabilities while concurrently reducing internal capabilities for expediency's sake. As we continue to reform and develop our national intelligence capabilities, the last thing we need to do is step out along that path across the IC..

Too late. DIA and CIA are heavily outsourced - well over 50% by some estimates.

Also, as Rex stated, and I thought was implied in my post, open source information isn't assessed in a vacuum - it is simply one of many sources of information that the analyst must absorb.

As I said to Rex, many analysts are already specialists, not generalists, and they like it that way. The agencies specialize as well (NRO, NSA, NGA, to name a few). It may not be the ideal, but that's how it is and it isn't changing anytime soon.

The "acceptance" of the value of OSINT within the IC is mainly a generational issue. Many "senior" guys are uncomfortable with the broad spectrum of open source media (both currently available and emerging) and prefer working with traditional sources, while the young'uns are perfectly capable of working in both worlds and of effectively integrating the two.

Agreed, and we don't have the time to wait for that "generational issue" to cease being an issue, which means that it's currently a problem which isn't going away.

I'm not suggesting that is a permanent fix, however considering how far behind we are, we need to run, not walk, towards getting up to speed to confront the variety of technological threats that threaten the U.S., and that are growing daily and that fact should compel us to keep an open mind as to where solutions can be found.

[Jedburgh]

IMINT, MASINT and SIGINT require a degree of specific technical expertise other than the relatively straightforward tradecraft of intelligence analysis for analysis of the raw product. OSINT does not. There is zero need for a specific "OSINT analysis" discipline; as I stated before, it falls easily within the skillset of the traditional intelligence analyst.

However, speaking to assertions of interdisciplinary rivalries, in my personal experience I have not seen any general trend of professionals in those fields "looking down their noses" at the all-source analyst. Certain individuals, perhaps, but not a general attitude. On the contrary, In my previous life as a HUMINT collector, I have often personally worked with IMINT, SIGINT and all-source analysts across a spectrum of operational missions and never had any issues. (I can't say anything about the MASINT guys, though) At times we've all worked together as a multi-disciplined body to address specific issues and it always worked well. The rivalries that do pop up are generally good natured and positive in that they drive competititon to succeed. The real problems that do exist are at a higher interagency level, where very senior egos drive the turf battles.

Yes, several agencies are already involved in outsourcing. However, rather than calling up someone else to cut the grass while sitting down with a beer to watch football, those with power to influence decisions should be exerting their efforts to halt that process and to expedite the necessary internal restructuring. Short-term expedient solutions feed into long-term strategic dangers. By outsourcing to a growing degree we are set on the road to committing strategic damage to our national intelligence capabilities. We need to stop the parasites who continue to lobby for increasing this trend. Fear-mongering to derail carefully thought-out restructuring in favor of quick fix solutions is a common tactic. Their goals are not national security, but personal profit.

The real long-term solution, as I've stated several times before, is in personnel management. Hire good people, train and mentor them well, ensure that they have regular opportunities to travel within their areas of focus (for those who aren't already regionally-based) and have in place functional professional-development programs that ensures they keep up with emerging technology and methodology within their respective fields.

[JeffC]

Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.

The CRS study that you linked to points out some of the problems that exist within the IC as regards OSINT. Pages 8 and 9 list a few of the major obstacles, but the problems extend beyond this to a lack of confidence in the actual value of OSINT by some senior members of the effected agencies:

As far as why regular analytic production should not be outsourced - no, the reasons for that aren't obvious. In fact, the high percentage of outsourcing at the CIA has been the subject of a lot of debate. What do you think are the reasons for not outsourcing OSINT collection and analysis?