Page 1 of 3 123 LastLast
Results 1 to 20 of 78

Thread: Open Source Intelligence (OSINT): merged thread

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Small Wars Journal SWJED's Avatar
    Join Date
    Sep 2005
    Location
    Largo, Florida
    Posts
    3,989

    Default Open Source Intelligence (OSINT): merged thread

    9 Nov. New York Times - Intelligence Center Is Created for Unclassified Information. Excerpt follows:

    "Top intelligence officials announced on Tuesday the creation of a new agency, the Open Source Center, to gather and analyze information from the Web, broadcasts, newspapers and other unclassified sources around the world."

    "The premise of the center, announced as part of the restructuring of the nation's intelligence agencies by the director of national intelligence, John D. Negroponte, is that some critical information to understand threats to national security requires neither spies nor satellites to collect."

    "This 'open source' information can include anything from sermons broadcast from radical mosques in the Middle East to reports in the provincial Chinese press of possible avian flu outbreaks. Such material has often been undervalued by government policymakers, in part because it lacks the cachet of information gathered by more sensitive methods, intelligence officials said..."

    Moderator's Note

    Six threads have been merged here. Note there is separate one on Google Earth, which remains untouched.
    Last edited by davidbfpo; 01-13-2013 at 12:29 AM. Reason: Add note

  2. #2
    Registered User
    Join Date
    Nov 2005
    Location
    Kent, WA
    Posts
    9

    Default

    Re-inventing the Early Bird? Or institutionalizing it?
    (And is it true that the Early Bird has been politicized in the last 5 years?)

  3. #3
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    Quote Originally Posted by KenDawe
    Re-inventing the Early Bird? Or institutionalizing it?
    (And is it true that the Early Bird has been politicized in the last 5 years?)
    A bit more than the Early Bird.
    Quote Originally Posted by DNI
    In executing the National Strategy for Open Source, the Center will be a comprehensive Intelligence Community Center focused on all open sources, to include the Internet, databases, press, radio, television, video, geospatial data, photos, and commercial imagery. The Center's functions will include collection/acquisition; open source analysis and research; centralized training; and facilitation of government-wide access and exploitation.
    Extensive details are available in a couple of documents that have been uploaded to AKO KC Intelligence Reference Files, for those with an account.

  4. #4
    DDilegge
    Guest

    Default Neither...

    Quote Originally Posted by KenDawe
    Re-inventing the Early Bird? Or institutionalizing it?
    (And is it true that the Early Bird has been politicized in the last 5 years?)
    I read the Early Bird (Current News) every morning. It is simply coverage of major news items and Op-Ed pieces that address issues associated with the DoD. No opinion is offered, simply copies of news articles. While one might argue that the Early Bird only posts articles deemed "favorable", I would argue the opposite (for the most part) as I have seen news and opinions that slam the DoD on certain issues. The purpose here is to inform DoD leadership and others on what is being reported in the mainstream news media.

    One criticism I have of the Early Bird is that they did away with the Supplement several months ago. The supplement offered a whole slew of additional reporting that might not be deemed "headline news" but otherwise offered deeper insight and background on important issues. The reason given for dropping the supplement was required manpower resources so the main Early Bird could go from a five-day to a seven-day format.

    All that said, the SWJ tries to fill in and add depth to the Early Bird by posting Daily News Links. If you look at the address line (url) you can see our take-off (inside joke) on the Early Bird....

    Referencing the new Open Source Intelligence Center - whole different ballgame - it will (or should) mine all open source data - not just headline news. Moreover, it will add analysis, not simply a regurgitation...

  5. #5
    Small Wars Journal SWJED's Avatar
    Join Date
    Sep 2005
    Location
    Largo, Florida
    Posts
    3,989

    Default CIA Mines 'Rich' Content From Blogs

    19 April Washington Times - CIA Mines 'Rich' Content From Blogs.

    ... The new Open Source Center (OSC) at CIA headquarters recently stepped up data collection and analysis based on bloggers worldwide and is developing new methods to gauge the reliability of the content, said OSC Director Douglas J. Naquin.

    "A lot of blogs now have become very big on the Internet, and we're getting a lot of rich information on blogs that are telling us a lot about social perspectives and everything from what the general feeling is to ... people putting information on there that doesn't exist anywhere else," Mr. Naquin told The Washington Times.

    Eliot A. Jardines, assistant deputy director of national intelligence for open source, said the amount of unclassified intelligence reaching Mr. Bush and senior policy-makers has increased as a result of the center's creation in November.

    "We're certainly scoring a number of wins with our ultimate customer," said Mr. Jardines, who became the first high-level official in charge of the government's nonsecret intelligence in December.

    "I can't get into detail of what, but I'll just say the amount of open source reporting that goes into the president's daily brief has gone up rather significantly," Mr. Jardines said. "There has been a real interest at the highest levels of our government, and we've been able to consistently deliver products that are on par with the rest of the intelligence community."...

    The OSC uses powerful computers and software technology to "sift" the Internet for valuable intelligence. It also buys information from commercial databases.

    In the past, open-source reports were used mainly by intelligence analysts...

  6. #6
    Small Wars Journal SWJED's Avatar
    Join Date
    Sep 2005
    Location
    Largo, Florida
    Posts
    3,989

    Default Seeking Iran Intelligence, U.S. Tries Google

    11 December Washington Post - Seeking Iran Intelligence, U.S. Tries Google by Dafna Linzer.

    When the State Department recently asked the CIA for names of Iranians who could be sanctioned for their involvement in a clandestine nuclear weapons program, the agency refused, citing a large workload and a desire to protect its sources and tradecraft.

    Frustrated, the State Department assigned a junior Foreign Service officer to find the names another way -- by using Google. Those with the most hits under search terms such as "Iran and nuclear," three officials said, became targets for international rebuke Friday when a sanctions resolution circulated at the United Nations.

    Policymakers and intelligence officials have always struggled when it comes to deciding how and when to disclose secret information, such as names of Iranians with suspected ties to nuclear weapons. In some internal debates, policymakers win out and intelligence is made public to further political or diplomatic goals. In other cases, such as this one, the intelligence community successfully argues that protecting information outweighs the desires of some to share it with the world.

    But that argument can also put the U.S. government in the awkward position of relying, in part, on an Internet search to select targets for international sanctions...

  7. #7
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    Sounds almost like another urban myth.

    Hell, even after the Agency refuses an RFI from State, State has its own Bureau of Intelligence and Research staffed with professional analysts. They are just as good as any analysts at the Agency, although there is a significant difference in type, capability, and availability of collection resources.

    I just don't see an RFI of consequence being filled by a cherry FSO in a cubicle when they have professional analysts on-hand. I see this story being leaked in this manner by someone at State as providing a public view of ongoing bureaucratic infighting.

  8. #8
    Council Member bismark17's Avatar
    Join Date
    Mar 2006
    Location
    Seattle, Wa
    Posts
    206

    Default

    My understanding is that State's Intel Unit used to be the Open Source Unit for the OSS. Once the OSS was disbanded they were placed with the State Department.

  9. #9
    Council Member Tom Odom's Avatar
    Join Date
    Oct 2005
    Location
    DeRidder LA
    Posts
    3,949

    Default State INR

    I can't speak to the OSS connection but my experience with State INR as a historian wading through the 1964 Congo Crisis, as an intelligence analyst in Gulf War 1, and as an intel operator in Zaire and Rwanda was that INR did first rate work.

    In many ways that work was possible because INR operated semi-independently of the regional bureaus and was not subject to policy pressures as are the bureaus and the embassies.

    Finally my associates at INR stayed in their roles longer and developed greater depth than the analysts at DIA or CIA.

    Tom

  10. #10
    Council Member CSC2005's Avatar
    Join Date
    Oct 2005
    Location
    VA
    Posts
    30

    Default Inr

    It was not by accident that the new head of analysis under the DNI was the former head of State INR. Despite having fewer analysts then the CIA has lunchroom workers, INR has always prided themselves on long term deep analysis by analysts who cover an area for decades. Instead of playing 5-year old soccer "all the crowd the ball" intel or try to be a classified CNN, INR focuses on quality of analysis. A trait the 9/11 report said was "lacking" at CIA and DIA.

  11. #11
    Moderator Steve Blair's Avatar
    Join Date
    Oct 2005
    Location
    Montana
    Posts
    3,195

    Default

    Quote Originally Posted by Jedburgh View Post
    Sounds almost like another urban myth.

    Hell, even after the Agency refuses an RFI from State, State has its own Bureau of Intelligence and Research staffed with professional analysts. They are just as good as any analysts at the Agency, although there is a significant difference in type, capability, and availability of collection resources.

    I just don't see an RFI of consequence being filled by a cherry FSO in a cubicle when they have professional analysts on-hand. I see this story being leaked in this manner by someone at State as providing a public view of ongoing bureaucratic infighting.
    I would tend to agree. This just sounds too "pat" to me. It could also be someone with the proverbial political ax to grind, since the Google spin would be a good way to discredit the accuracy of anyone's name who turned up in said memo.

  12. #12
    Small Wars Journal SWJED's Avatar
    Join Date
    Sep 2005
    Location
    Largo, Florida
    Posts
    3,989

    Default Army Seeks to Catalyze Open Source Intelligence

    From Secrecy News (FAS Blog) - Army Seeks to Catalyze Open Source Intelligence.

    A new U.S. Army Field Manual is intended to advance the development and use of open source intelligence (OSINT), which is intelligence that is derived from publicly available data legally obtained.

    "The value of publicly available information as a source of intelligence has... often been overlooked in Army intelligence operations. This manual (pdf) provides a catalyst for renewing the Army's awareness of the value of open sources; establishing a common understanding of OSINT; and developing systematic approaches to collection, processing, and analysis of publicly available information."

    The growing military appreciation of open source intelligence arises from the ever-increasing quality of public sources and the evident limitations of traditional classified approaches...

  13. #13
    Council Member bismark17's Avatar
    Join Date
    Mar 2006
    Location
    Seattle, Wa
    Posts
    206

    Default Manual Review

    This is a good manual. While there is a lot of jargon that is DOD or DA specific the content is very interesting and applicable to Domestic L.E. Intel. It seems like the Army FMs just keep getting better. They have come a long way from the old FM 7-8 or 7-70 manuals that I used to read.

    The manual is well laid out and organized. It's very relevant and useful for end users and not just "doctrine" specialists who are just skimming for the latest buzz words to throw out to impress the boss. It includes some very useful appendixes that add to the overall content. Interested readers who are not up on the PIR/SIR/IR methodology may find it useful to develop some knowledge of the Army Intel doctrine or process before digging in.

  14. #14
    Council Member marct's Avatar
    Join Date
    Aug 2006
    Location
    Ottawa, Canada
    Posts
    3,682

    Default

    Quote Originally Posted by bismark17 View Post
    Interested readers who are not up on the PIR/SIR/IR methodology may find it useful to develop some knowledge of the Army Intel doctrine or process before digging in.
    Any recommended readings on getting up to speed in he PIR/SIR/IR area?

    Marc
    Sic Bisquitus Disintegrat...
    Marc W.D. Tyrrell, Ph.D.
    Institute of Interdisciplinary Studies,
    Senior Research Fellow,
    The Canadian Centre for Intelligence and Security Studies, NPSIA
    Carleton University
    http://marctyrrell.com/

  15. #15
    Council Member Stan's Avatar
    Join Date
    Dec 2006
    Location
    Estonia
    Posts
    3,817

    Default DIA would be a good start

    Humint, we used to call it globally. That included OSINT, but only a tinny part of HUMINT....can't hardly wait for Tom to log-on with his two cents.

    OSINT, more or less what the more than 250 ARMY NCOs and Officers did for years (that would be 12 for me). Some were freebies at the local cocktail curcuits, others you had to work for.

    Without a suitable background and language, you're up against a difficult task. Written press has its own slang, and context can get way outta hand if you have no idea what they are saying.

    45 weeks in California does not get you there. Reading open source materials is one thing, but did you understand the overall context from say, an African's point of view ?

    Regards, Stan

  16. #16
    Council Member bismark17's Avatar
    Join Date
    Mar 2006
    Location
    Seattle, Wa
    Posts
    206

    Default Just an idea

    I learned a lot from a great friend over coffee just outside of the gates of Ft. Lewis. Until my eyes started drifting over to more interesting sights in the coffee shop......The new Army Intel Analysis manual is pretty good. FM34-3??? There is also a great one on Intel Analysis in Urban environments. I will send you the links.

    The great thing is all of this stuff is on the net. At the risk of making the military guys mad at me, I don't think the methodology is that critical. It's good and works well for them in their realms. But they have to do deal with a very complex variety of potential situations and areas of operation that civilians don't have to worry about.

    I don't have to worry about having a pre built process to set up an intel shop in Baghdad or North Korea, I just have to know what's going on in that barber shop where my players are hanging in as an example.

  17. #17
    Council Member BILL's Avatar
    Join Date
    Dec 2007
    Posts
    41

    Default YOU did OSINT for 12 years

    thank you santa: You are the most rare "OSINTithus" very rare breed of suspect authenticity. We really have OSINT guys? Ok that is both scary and hopeful at the same time. Hopeful because we maynot be as far behind as I thought. Scary concept! You read OSINT and did what? Our OSINT CAPABILITIES are awesome, impresses me continually. We have been active for 1 1/2 yrs. But we treat the "Trooper" like an infant baby. We outline his areas of Interest, names, concepts, an open feed he can change/adjust at will. We feed him a steady diet 24/7 of things of INTEREST to him. A big steady vacuume of the Internet, sites, RSS, news, blogs, forums, docs, pdfs, etc. We make him the expert on real time OSINT intel, in his area, on a very pragmatic level. And they chose and run the ops etc. The end user directs the feed, amount, quality, targets, definitions, etc. We have developed a new concept, "Paradigm Intelligence", a very fancy inductive reasoning system. We have found we can pull data from a closed cell with it, it is anthropology based, as it looks more at actions and less at words. I hope to get some sense of what you DID. Thanks Bill
    Quote Originally Posted by Stan View Post
    Humint, we used to call it globally. That included OSINT, but only a tinny part of HUMINT....can't hardly wait for Tom to log-on with his two cents.

    OSINT, more or less what the more than 250 ARMY NCOs and Officers did for years (that would be 12 for me). Some were freebies at the local cocktail curcuits, others you had to work for.

    Without a suitable background and language, you're up against a difficult task. Written press has its own slang, and context can get way outta hand if you have no idea what they are saying.

    45 weeks in California does not get you there. Reading open source materials is one thing, but did you understand the overall context from say, an African's point of view ?

    Regards, Stan

  18. #18
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    ...The growing military appreciation of open source intelligence arises from the ever-increasing quality of public sources and the evident limitations of traditional classified approaches...
    The new OSINT FM is only the latest in the evolution of the military approach to OSINT. Within the US Army we've had the INSCOM OSINT Handbook, published in May 03, and the Joint Military Intelligence Training Center Open Source Exploitation: A Guide for Intelligence Analysts, published in Apr 04 (this is the second edition, I never saw the first). Both had relatively limited distribution.

    As far as I know, NATO was ahead of the game with a formal manual published in Nov 01, the NATO OSINT Handbook and the NATO OSINT Reader, published in Feb 02.

    For those with AKO access, the OSINT Knowledge Center contains copies of the presentations from a number of intelligence assets and their approaches to OSINT as put out during a series of OSINT conferences held in 05 and 06.
    Last edited by Jedburgh; 01-26-2007 at 08:08 PM.

  19. #19
    Council Member Stan's Avatar
    Join Date
    Dec 2006
    Location
    Estonia
    Posts
    3,817

    Default The Coffee Shop

    I learned a lot from a great friend over coffee just outside of the gates of Ft. Lewis. Until my eyes started drifting over to more interesting sights in the coffee shop.
    Hi Bismark !
    That's not too far fetched. Most of Humint today is little more than common sense and eyes open. DIAs course goes farther with lessons and experiences over the last 25 years. That "book" at AKO is much the same. The folks that put that together are former Army NCOs and Officers who once worked for DIA.

    Regards, Stan

  20. #20
    Council Member
    Join Date
    Nov 2005
    Posts
    13

    Default

    We used to joke that OSINT was just an excuse to surf the niprnet, check email, buy crap from amazon, etc.

    We started to get reports that some bad guys in the area were going to meet with a reporter, and we were tracking that, trying to find out where and when.

    We missed the meet, but got lots of reports about it after it happened. Sure enough a few days later an analyst from higher emails us some pics from Reuters showing a group of BG's meeting in a typical sit down. Weapons, unifroms ak's, commo etc. The main BG's had their faces covered but some did not.

    OSINT, actually proved useful, and will continue to do so.

    This is especially true considering the BG's effective use of IO, along with the "good" of getting their message, videos, etc out there they get increased exposure and may end up showing us something we need to know.

Similar Threads

  1. Intelligence: failures, gaps and knowledge gaps
    By SWJED in forum Intelligence
    Replies: 5
    Last Post: 01-04-2017, 03:29 PM
  2. The David Kilcullen Collection (merged thread)
    By Fabius Maximus in forum Doctrine & TTPs
    Replies: 451
    Last Post: 03-31-2016, 03:23 PM
  3. Open Source Analysis of WikiLeaks?
    By BobKing in forum Intelligence
    Replies: 15
    Last Post: 09-11-2010, 04:08 PM
  4. Replies: 0
    Last Post: 11-25-2008, 10:28 PM
  5. Terrorist Groups Thesaurus / Open Source Guide
    By SWJED in forum Adversary / Threat
    Replies: 1
    Last Post: 11-17-2006, 03:19 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •