Open Source Intelligence (OSINT): merged thread

9 Nov. New York Times - Intelligence Center Is Created for Unclassified Information. Excerpt follows:

"Top intelligence officials announced on Tuesday the creation of a new agency, the Open Source Center, to gather and analyze information from the Web, broadcasts, newspapers and other unclassified sources around the world."

"The premise of the center, announced as part of the restructuring of the nation's intelligence agencies by the director of national intelligence, John D. Negroponte, is that some critical information to understand threats to national security requires neither spies nor satellites to collect."

"This 'open source' information can include anything from sermons broadcast from radical mosques in the Middle East to reports in the provincial Chinese press of possible avian flu outbreaks. Such material has often been undervalued by government policymakers, in part because it lacks the cachet of information gathered by more sensitive methods, intelligence officials said..."

Moderator's Note

Six threads have been merged here. Note there is separate one on Google Earth, which remains untouched.

Re-inventing the Early Bird? Or institutionalizing it?
(And is it true that the Early Bird has been politicized in the last 5 years?)

Quote:

---

Originally Posted by KenDawe

Re-inventing the Early Bird? Or institutionalizing it?
(And is it true that the Early Bird has been politicized in the last 5 years?)

---

A bit more than the Early Bird.

Quote:

---

Originally Posted by DNI

In executing the National Strategy for Open Source, the Center will be a comprehensive Intelligence Community Center focused on all open sources, to include the Internet, databases, press, radio, television, video, geospatial data, photos, and commercial imagery. The Center's functions will include collection/acquisition; open source analysis and research; centralized training; and facilitation of government-wide access and exploitation.

---

Extensive details are available in a couple of documents that have been uploaded to AKO KC Intelligence Reference Files, for those with an account.

Quote:

---

Originally Posted by KenDawe

Re-inventing the Early Bird? Or institutionalizing it?
(And is it true that the Early Bird has been politicized in the last 5 years?)

---

I read the Early Bird (Current News) every morning. It is simply coverage of major news items and Op-Ed pieces that address issues associated with the DoD. No opinion is offered, simply copies of news articles. While one might argue that the Early Bird only posts articles deemed "favorable", I would argue the opposite (for the most part) as I have seen news and opinions that slam the DoD on certain issues. The purpose here is to inform DoD leadership and others on what is being reported in the mainstream news media.

One criticism I have of the Early Bird is that they did away with the Supplement several months ago. The supplement offered a whole slew of additional reporting that might not be deemed "headline news" but otherwise offered deeper insight and background on important issues. The reason given for dropping the supplement was required manpower resources so the main Early Bird could go from a five-day to a seven-day format.

All that said, the SWJ tries to fill in and add depth to the Early Bird by posting Daily News Links. If you look at the address line (url) you can see our take-off (inside joke) on the Early Bird....

Referencing the new Open Source Intelligence Center - whole different ballgame - it will (or should) mine all open source data - not just headline news. Moreover, it will add analysis, not simply a regurgitation...

19 April Washington Times - CIA Mines 'Rich' Content From Blogs.

Quote:

---

... The new Open Source Center (OSC) at CIA headquarters recently stepped up data collection and analysis based on bloggers worldwide and is developing new methods to gauge the reliability of the content, said OSC Director Douglas J. Naquin.

"A lot of blogs now have become very big on the Internet, and we're getting a lot of rich information on blogs that are telling us a lot about social perspectives and everything from what the general feeling is to ... people putting information on there that doesn't exist anywhere else," Mr. Naquin told The Washington Times.

Eliot A. Jardines, assistant deputy director of national intelligence for open source, said the amount of unclassified intelligence reaching Mr. Bush and senior policy-makers has increased as a result of the center's creation in November.

"We're certainly scoring a number of wins with our ultimate customer," said Mr. Jardines, who became the first high-level official in charge of the government's nonsecret intelligence in December.

"I can't get into detail of what, but I'll just say the amount of open source reporting that goes into the president's daily brief has gone up rather significantly," Mr. Jardines said. "There has been a real interest at the highest levels of our government, and we've been able to consistently deliver products that are on par with the rest of the intelligence community."...

The OSC uses powerful computers and software technology to "sift" the Internet for valuable intelligence. It also buys information from commercial databases.

In the past, open-source reports were used mainly by intelligence analysts...

---

11 December Washington Post - Seeking Iran Intelligence, U.S. Tries Google by Dafna Linzer.

Quote:

---

When the State Department recently asked the CIA for names of Iranians who could be sanctioned for their involvement in a clandestine nuclear weapons program, the agency refused, citing a large workload and a desire to protect its sources and tradecraft.

Frustrated, the State Department assigned a junior Foreign Service officer to find the names another way -- by using Google. Those with the most hits under search terms such as "Iran and nuclear," three officials said, became targets for international rebuke Friday when a sanctions resolution circulated at the United Nations.

Policymakers and intelligence officials have always struggled when it comes to deciding how and when to disclose secret information, such as names of Iranians with suspected ties to nuclear weapons. In some internal debates, policymakers win out and intelligence is made public to further political or diplomatic goals. In other cases, such as this one, the intelligence community successfully argues that protecting information outweighs the desires of some to share it with the world.

But that argument can also put the U.S. government in the awkward position of relying, in part, on an Internet search to select targets for international sanctions...

---

Sounds almost like another urban myth.

Hell, even after the Agency refuses an RFI from State, State has its own Bureau of Intelligence and Research staffed with professional analysts. They are just as good as any analysts at the Agency, although there is a significant difference in type, capability, and availability of collection resources.

I just don't see an RFI of consequence being filled by a cherry FSO in a cubicle when they have professional analysts on-hand. I see this story being leaked in this manner by someone at State as providing a public view of ongoing bureaucratic infighting.

My understanding is that State's Intel Unit used to be the Open Source Unit for the OSS. Once the OSS was disbanded they were placed with the State Department.

Quote:

---

Originally Posted by Jedburgh

Sounds almost like another urban myth.

Hell, even after the Agency refuses an RFI from State, State has its own Bureau of Intelligence and Research staffed with professional analysts. They are just as good as any analysts at the Agency, although there is a significant difference in type, capability, and availability of collection resources.

I just don't see an RFI of consequence being filled by a cherry FSO in a cubicle when they have professional analysts on-hand. I see this story being leaked in this manner by someone at State as providing a public view of ongoing bureaucratic infighting.

---

I would tend to agree. This just sounds too "pat" to me. It could also be someone with the proverbial political ax to grind, since the Google spin would be a good way to discredit the accuracy of anyone's name who turned up in said memo.

I can't speak to the OSS connection but my experience with State INR as a historian wading through the 1964 Congo Crisis, as an intelligence analyst in Gulf War 1, and as an intel operator in Zaire and Rwanda was that INR did first rate work.

In many ways that work was possible because INR operated semi-independently of the regional bureaus and was not subject to policy pressures as are the bureaus and the embassies.

Finally my associates at INR stayed in their roles longer and developed greater depth than the analysts at DIA or CIA.

Tom

It was not by accident that the new head of analysis under the DNI was the former head of State INR. Despite having fewer analysts then the CIA has lunchroom workers, INR has always prided themselves on long term deep analysis by analysts who cover an area for decades. Instead of playing 5-year old soccer "all the crowd the ball" intel or try to be a classified CNN, INR focuses on quality of analysis. A trait the 9/11 report said was "lacking" at CIA and DIA.

How has INR fared in the big intelligence shuffle of the past five years?

These guys sound like people I'd like to work for...

For what it is worth I think INR is the cream of the crop for intel analysis. They do not rely on billion dollar collection systems or having thousands of analysts, instead they focus on history, culture, language and in-country experience. Their products are focused toward the diplomatic leadership, but they are first rate for deep analysis. The only problem is that many foreign service officers, who run DoS, think they are better than intel analysts and INR gets second billing in the very political senior levels of the DoS.

Those are just my two cents from working with then. The big agencies have alot to learn form INR. I welcome any other thoughts.

Quote:

---

Originally Posted by CSC2005

For what it is worth I think INR is the cream of the crop for intel analysis. They do not rely on billion dollar collection systems or having thousands of analysts, instead they focus on history, culture, language and in-country experience. Their products are focused toward the diplomatic leadership, but they are first rate for deep analysis. The only problem is that many foreign service officers, who run DoS, think they are better than intel analysts and INR gets second billing in the very political senior levels of the DoS.

Those are just my two cents from working with then. The big agencies have alot to learn form INR. I welcome any other thoughts.

---

Art,

Though I am now over 7 years removed from the IC, my impressions in the 80's and 90's echo your observations. Whenever an INR analyst spoke at a NIE or any other gathering of community analysts I listened because there was meat behind what they had to say - and like you said - based on history, culture, language and in-country experience.

Dave

From Secrecy News (FAS Blog) - Army Seeks to Catalyze Open Source Intelligence.

Quote:

---

A new U.S. Army Field Manual is intended to advance the development and use of open source intelligence (OSINT), which is intelligence that is derived from publicly available data legally obtained.

"The value of publicly available information as a source of intelligence has... often been overlooked in Army intelligence operations. This manual (pdf) provides a catalyst for renewing the Army's awareness of the value of open sources; establishing a common understanding of OSINT; and developing systematic approaches to collection, processing, and analysis of publicly available information."

The growing military appreciation of open source intelligence arises from the ever-increasing quality of public sources and the evident limitations of traditional classified approaches...

---

This is a good manual. While there is a lot of jargon that is DOD or DA specific the content is very interesting and applicable to Domestic L.E. Intel. It seems like the Army FMs just keep getting better. They have come a long way from the old FM 7-8 or 7-70 manuals that I used to read.

The manual is well laid out and organized. It's very relevant and useful for end users and not just "doctrine" specialists who are just skimming for the latest buzz words to throw out to impress the boss. It includes some very useful appendixes that add to the overall content. Interested readers who are not up on the PIR/SIR/IR methodology may find it useful to develop some knowledge of the Army Intel doctrine or process before digging in.

Quote:

---

Originally Posted by bismark17

Interested readers who are not up on the PIR/SIR/IR methodology may find it useful to develop some knowledge of the Army Intel doctrine or process before digging in.

---

Any recommended readings on getting up to speed in he PIR/SIR/IR area?

Marc

Humint, we used to call it globally. That included OSINT, but only a tinny part of HUMINT....can't hardly wait for Tom to log-on with his two cents.

OSINT, more or less what the more than 250 ARMY NCOs and Officers did for years (that would be 12 for me). Some were freebies at the local cocktail curcuits, others you had to work for.

Without a suitable background and language, you're up against a difficult task. Written press has its own slang, and context can get way outta hand if you have no idea what they are saying.

45 weeks in California does not get you there. Reading open source materials is one thing, but did you understand the overall context from say, an African's point of view ?

Regards, Stan

I learned a lot from a great friend over coffee just outside of the gates of Ft. Lewis. Until my eyes started drifting over to more interesting sights in the coffee shop......The new Army Intel Analysis manual is pretty good. FM34-3??? There is also a great one on Intel Analysis in Urban environments. I will send you the links.

The great thing is all of this stuff is on the net. At the risk of making the military guys mad at me, I don't think the methodology is that critical. It's good and works well for them in their realms. But they have to do deal with a very complex variety of potential situations and areas of operation that civilians don't have to worry about.

I don't have to worry about having a pre built process to set up an intel shop in Baghdad or North Korea, I just have to know what's going on in that barber shop where my players are hanging in as an example.

Hi John,

Quote:

---

Originally Posted by bismark17

I learned a lot from a great friend over coffee just outside of the gates of Ft. Lewis. Until my eyes started drifting over to more interesting sights in the coffee shop......The new Army Intel Analysis manual is pretty good. FM34-3??? There is also a great one on Intel Analysis in Urban environments. I will send you the links.

---

Thanks, I'd appreciate it.

Quote:

---

Originally Posted by bismark17

The great thing is all of this stuff is on the net. At the risk of making the military guys mad at me, I don't think the methodology is that critical. It's good and works well for them in their realms. But they have to do deal with a very complex variety of potential situations and areas of operation that civilians don't have to worry about.

I don't have to worry about having a pre built process to set up an intel shop in Baghdad or North Korea, I just have to know what's going on in that barber shop where my players are hanging in as an example.

---

Good point. Actually, I'm also wondering how much of it can be adapted for Anthropological fieldwork. I can just see the reactons ofsome of my colleagues when I assign a US Army manual as a textbook :D.

Marc