Page 3 of 4 FirstFirst 1234 LastLast
Results 41 to 60 of 78

Thread: Open Source Intelligence (OSINT): merged thread

  1. #41
    Banned
    Join Date
    Jun 2007
    Location
    Poulsbo, WA
    Posts
    252

    Default

    Quote Originally Posted by Jedburgh View Post
    Your statement implies that the IC does not have the skills, people or agility to conduct effective analysis of open sources. I beg to differ. (although I would definitely state that neither DHS nor the Bureau have those resources - they are both still struggling to develop their emerging intel capabilities into something approaching minimally capable)

    In sum, any properly trained and experienced intelligence professional is fully capable of effectively exploiting open source material. The fundamental tradecraft of the intelligence analyst remains the same whether the material is unclass or on the high side.

    The primary struggle is with collection. Given the overwhelming amount of information available through a broad spectrum of media in a babble of languages, dialects and local jargon, collection structures have yet to meet the needs of analysts serving agencies with widely differing collection priorities. That problem is critical enough to justify contracting out certain specific types of collection.

    However, although analytic products obtained from third party vendors are also valuable information sources (and the government has long used such vendors to conduct special studies), regular analytic production should not be outsourced - for what should be very obvious reasons.
    The CRS study that you linked to points out some of the problems that exist within the IC as regards OSINT. Pages 8 and 9 list a few of the major obstacles, but the problems extend beyond this to a lack of confidence in the actual value of OSINT by some senior members of the effected agencies:

    As far as why regular analytic production should not be outsourced - no, the reasons for that aren't obvious. In fact, the high percentage of outsourcing at the CIA has been the subject of a lot of debate. What do you think are the reasons for not outsourcing OSINT collection and analysis?

  2. #42
    Banned
    Join Date
    Jun 2007
    Location
    Poulsbo, WA
    Posts
    252

    Default

    Quote Originally Posted by Rex Brynen View Post
    I completely agree with Ted.

    I would also add--as someone who works with both OSINT as well as very high end classified material, that I think its very important that the streams not be too separated, since one is often needed to make sense of the other (certainly for the kind of assessment work I've done).
    The CRS report that Ted linked to provides multiple reasons for doubt to be cast on the ability of the IC to fully take advantage of OSINT. The history of Intelligence analysis is a spotted one, so I'm not clear as to why Ted or anyone would think that analysis and collection of OSINT cannot be outsourced for better effect. Since you agree with Ted, perhaps you'll share your reasons?

  3. #43
    Council Member
    Join Date
    Aug 2007
    Location
    Montreal
    Posts
    1,600

    Default

    Quote Originally Posted by JeffC View Post
    The CRS report that Ted linked to provides multiple reasons for doubt to be cast on the ability of the IC to fully take advantage of OSINT. The history of Intelligence analysis is a spotted one, so I'm not clear as to why Ted or anyone would think that analysis and collection of OSINT cannot be outsourced for better effect. Since you agree with Ted, perhaps you'll share your reasons?
    I should start by saying the role of OSINT in analysis, the sorts of OSINT we're talking about, the technical challenges of collection and analysis, and its relative weight in production, all vary dramatically depending on the sort of analysis that one is doing. I can really only reflect on its role in the kind of (political and strategic assessment) work that I've done.

    Second, it should be noted that there is already a lot of OSINT collection and analysis built into the system on the government side: a good share of diplomatic reporting, for example, is essentially OSINT collection and analysis. That reporting, in turn, feeds into IC product.

    My concern about too much outsourcing of OSINT analysis stems from the observation that it is sometimes access to the high end classified material that shapes the relevance of OSINT, and those without access to this aren't always in a good position to understand the significance of what they are seeing in the OSINT material. I must admit, I always like to see data as raw as possible, to determine that the analytical conclusions drawn by others are the same ones that I would also draw.

    I'm also worried about a byproduct of too much OSINT outsourcing being the implicit development of a division of labour--"you do the OSINT, we'll do the COMINT/HUMINT/etc"--which runs counter to the fundamental need for analysts to use the full spectrum of material.

    I should add that my concerns are much less applicable to the outsourcing of some aspects of collection (which, as noted, is already done), and "preanalysis" of this material of the "hey, this is especially interesting" flag, or the "this relates in interesting ways to previous material" flag.

    I have to dash to a conference in 2 minutes, so this answer rambles far more than I would wish. I'm sure, in any event, that Ted will provide a far more thoughtful and coherent answer than I'm capable of!

  4. #44
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    The quick answer is that contracting out capabilities that should be developed internally weakens rather than strengthens our national intelligence structure.

    We've already done long-term damage to military intelligence by "outsourcing" critical capabilities while concurrently reducing internal capabilities for expediency's sake. As we continue to reform and develop our national intelligence capabilities, the last thing we need to do is step out along that path across the IC.

    Also, as Rex stated, and I thought was implied in my post, open source information isn't assessed in a vacuum - it is simply one of many sources of information that the analyst must absorb. That is why I said that outsourcing of collection is justified (but only for the short term, until we effectively modify and develop effective internal structures within the IC) but not the outsourcing of analysis.

    The "acceptance" of the value of OSINT within the IC is mainly a generational issue. Many "senior" guys are uncomfortable with the broad spectrum of open source media (both currently available and emerging) and prefer working with traditional sources, while the young'uns are perfectly capable of working in both worlds and of effectively integrating the two.

    Myself, I find the view of those old boys kind of odd, as OSINT has long played a critical role in strategic intelligence analysis. I'm sure there's a couple of Cold Warriors on the board who recall the effort put into anayzing and interpreting articles published in the official Soviet media to support assessments of powershifts and policy changes in the Kremlin.

  5. #45
    i pwnd ur ooda loop selil's Avatar
    Join Date
    Sep 2006
    Location
    Belly of the beast
    Posts
    2,112

    Default

    As has been said the OSINT is but one collection mechanism among many. I take issue with outsourcing it or further diluting the variety information in consideration, but more importantly I take issue with creating another non-permeable barrier in the collection process. If I had to put my finger on the issues with intelligence analysis and processing it is not the volume of information or the speed it is the number of analysts looking at any one area at a time. We need more highly trained analysts inside the intelligence organizations with better tools to do their job better. Outsourcing is not the answer it is only a band-aid over leprosy.
    Sam Liles
    Selil Blog
    Don't forget to duck Secret Squirrel
    The scholarship of teaching and learning results in equal hatred from latte leftists and cappuccino conservatives.
    All opinions are mine and may or may not reflect those of my employer depending on the chance it might affect funding, politics, or the setting of the sun. As such these are my opinions you can get your own.

  6. #46
    Council Member
    Join Date
    Nov 2007
    Location
    Ocean Township, NJ
    Posts
    95

    Default

    Quote Originally Posted by Jedburgh View Post
    Myself, I find the view of those old boys kind of odd, as OSINT has long played a critical role in strategic intelligence analysis. I'm sure there's a couple of Cold Warriors on the board who recall the effort put into anayzing and interpreting articles published in the official Soviet media to support assessments of powershifts and policy changes in the Kremlin.
    To make the elders here feel really old...My advisor at Scranton was (in 61-62) a Soviet analyst for the CIA.

    Old-fashioned Kremlinology was often talked about, in a "for those of you who were too young to notice" fashion, in classes. Basically, as history.

    Example which sticks out: The immense effort put into getting...photos of the reviewing stands for the May Day and October Revolution parades. Hats or no hats, the positioning of various people, whether they were sitting or standing, analyzed to seemingly-absurd lengths.

    How much of that actually produced insights of value, might I ask my elders?

    *whistles the Soviet National Anthem boredly*

    (I'm young enough to say that...I miss the Cold War. When things were simple, and you knew your enemy was at least sane (and could even be worked with on occasion!). Yes, this points out my youth.)

  7. #47
    Banned
    Join Date
    Jun 2007
    Location
    Poulsbo, WA
    Posts
    252

    Default

    Quote Originally Posted by Rex Brynen View Post
    My concern about too much outsourcing of OSINT analysis stems from the observation that it is sometimes access to the high end classified material that shapes the relevance of OSINT, and those without access to this aren't always in a good position to understand the significance of what they are seeing in the OSINT material. I must admit, I always like to see data as raw as possible, to determine that the analytical conclusions drawn by others are the same ones that I would also draw.
    --- In the private sector, this is a given. Evidence is sourced, and anyone who cares to can examine the original source. It's only in the IC that source data is closely guarded rather than shared. It's a good example of one of many flaws in the present system.

    I'm also worried about a byproduct of too much OSINT outsourcing being the implicit development of a division of labour--"you do the OSINT, we'll do the COMINT/HUMINT/etc"--which runs counter to the fundamental need for analysts to use the full spectrum of material.
    But the current system IS a division of labor. There are MASINT analysts, SIGINT analysts, IMINT analysts, etc., who like the fact that they're specialists and look down their noses at All-Source analysts as nothing more than writers. There's a MITRE study that explores this and recommends some changes be made, but as far as I know, those changes haven't happened yet, and they may never happen at all. Additonally, there are specialist agencies: NRO, NSA, NGA that only focus on certain types of INT collection and analysis.

  8. #48
    Banned
    Join Date
    Jun 2007
    Location
    Poulsbo, WA
    Posts
    252

    Default

    Quote Originally Posted by Jedburgh View Post
    The quick answer is that contracting out capabilities that should be developed internally weakens rather than strengthens our national intelligence structure.

    We've already done long-term damage to military intelligence by "outsourcing" critical capabilities while concurrently reducing internal capabilities for expediency's sake. As we continue to reform and develop our national intelligence capabilities, the last thing we need to do is step out along that path across the IC..

    Too late. DIA and CIA are heavily outsourced - well over 50% by some estimates.


    Also, as Rex stated, and I thought was implied in my post, open source information isn't assessed in a vacuum - it is simply one of many sources of information that the analyst must absorb.
    As I said to Rex, many analysts are already specialists, not generalists, and they like it that way. The agencies specialize as well (NRO, NSA, NGA, to name a few). It may not be the ideal, but that's how it is and it isn't changing anytime soon.


    The "acceptance" of the value of OSINT within the IC is mainly a generational issue. Many "senior" guys are uncomfortable with the broad spectrum of open source media (both currently available and emerging) and prefer working with traditional sources, while the young'uns are perfectly capable of working in both worlds and of effectively integrating the two.
    Agreed, and we don't have the time to wait for that "generational issue" to cease being an issue, which means that it's currently a problem which isn't going away.

    I'm not suggesting that is a permanent fix, however considering how far behind we are, we need to run, not walk, towards getting up to speed to confront the variety of technological threats that threaten the U.S., and that are growing daily and that fact should compel us to keep an open mind as to where solutions can be found.

  9. #49
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    IMINT, MASINT and SIGINT require a degree of specific technical expertise other than the relatively straightforward tradecraft of intelligence analysis for analysis of the raw product. OSINT does not. There is zero need for a specific "OSINT analysis" discipline; as I stated before, it falls easily within the skillset of the traditional intelligence analyst.

    However, speaking to assertions of interdisciplinary rivalries, in my personal experience I have not seen any general trend of professionals in those fields "looking down their noses" at the all-source analyst. Certain individuals, perhaps, but not a general attitude. On the contrary, In my previous life as a HUMINT collector, I have often personally worked with IMINT, SIGINT and all-source analysts across a spectrum of operational missions and never had any issues. (I can't say anything about the MASINT guys, though) At times we've all worked together as a multi-disciplined body to address specific issues and it always worked well. The rivalries that do pop up are generally good natured and positive in that they drive competititon to succeed. The real problems that do exist are at a higher interagency level, where very senior egos drive the turf battles.

    Yes, several agencies are already involved in outsourcing. However, rather than calling up someone else to cut the grass while sitting down with a beer to watch football, those with power to influence decisions should be exerting their efforts to halt that process and to expedite the necessary internal restructuring. Short-term expedient solutions feed into long-term strategic dangers. By outsourcing to a growing degree we are set on the road to committing strategic damage to our national intelligence capabilities. We need to stop the parasites who continue to lobby for increasing this trend. Fear-mongering to derail carefully thought-out restructuring in favor of quick fix solutions is a common tactic. Their goals are not national security, but personal profit.

    The real long-term solution, as I've stated several times before, is in personnel management. Hire good people, train and mentor them well, ensure that they have regular opportunities to travel within their areas of focus (for those who aren't already regionally-based) and have in place functional professional-development programs that ensures they keep up with emerging technology and methodology within their respective fields.

  10. #50
    Council Member BILL's Avatar
    Join Date
    Dec 2007
    Posts
    41

    Default YOU did OSINT for 12 years

    thank you santa: You are the most rare "OSINTithus" very rare breed of suspect authenticity. We really have OSINT guys? Ok that is both scary and hopeful at the same time. Hopeful because we maynot be as far behind as I thought. Scary concept! You read OSINT and did what? Our OSINT CAPABILITIES are awesome, impresses me continually. We have been active for 1 1/2 yrs. But we treat the "Trooper" like an infant baby. We outline his areas of Interest, names, concepts, an open feed he can change/adjust at will. We feed him a steady diet 24/7 of things of INTEREST to him. A big steady vacuume of the Internet, sites, RSS, news, blogs, forums, docs, pdfs, etc. We make him the expert on real time OSINT intel, in his area, on a very pragmatic level. And they chose and run the ops etc. The end user directs the feed, amount, quality, targets, definitions, etc. We have developed a new concept, "Paradigm Intelligence", a very fancy inductive reasoning system. We have found we can pull data from a closed cell with it, it is anthropology based, as it looks more at actions and less at words. I hope to get some sense of what you DID. Thanks Bill
    Quote Originally Posted by Stan View Post
    Humint, we used to call it globally. That included OSINT, but only a tinny part of HUMINT....can't hardly wait for Tom to log-on with his two cents.

    OSINT, more or less what the more than 250 ARMY NCOs and Officers did for years (that would be 12 for me). Some were freebies at the local cocktail curcuits, others you had to work for.

    Without a suitable background and language, you're up against a difficult task. Written press has its own slang, and context can get way outta hand if you have no idea what they are saying.

    45 weeks in California does not get you there. Reading open source materials is one thing, but did you understand the overall context from say, an African's point of view ?

    Regards, Stan

  11. #51
    Council Member Stan's Avatar
    Join Date
    Dec 2006
    Location
    Estonia
    Posts
    3,817

    Default

    Quote Originally Posted by BILL View Post
    thank you santa: You are the most rare "OSINTithus" very rare breed of suspect authenticity. We really have OSINT guys? Ok that is both scary and hopeful at the same time. Hopeful because we maynot be as far behind as I thought. Scary concept! You read OSINT and did what?
    Bill,
    First and foremost, welcome to the SWC!
    One normally does not commence with personal attacks and skepticism for his/her first post. As time permits you, please go here and introduce yourself.

    Yes, Bill, we really did, and do have OSINT ‘guys’ out there. Forgive me if I don’t feel good about sharing my past and training with someone who has ‘N/A’ listed as his/her background in the user profile.

    Quote Originally Posted by BILL View Post
    Our OSINT CAPABILITIES are awesome, impresses me continually. We have been active for 1 1/2 yrs. But we treat the "Trooper" like an infant baby. We outline his areas of Interest, names, concepts, an open feed he can change/adjust at will. We feed him a steady diet 24/7 of things of INTEREST to him. A big steady vacuume of the Internet, sites, RSS, news, blogs, forums, docs, pdfs, etc. We make him the expert on real time OSINT intel, in his area, on a very pragmatic level. And they chose and run the ops etc. The end user directs the feed, amount, quality, targets, definitions, etc. Thanks Bill
    I was not referring to our typical soldiers in non-intelligence roles and capacities; rather intel-related MOSs and being poorly prepared to do same. My initial intelligence training is more than two decades old, so I’ll assume there have been significant advances in OSINT (such as the internet, blogs and forums).

    Regards, Stan

  12. #52
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    CSS, 9 Apr 08: Open Source Intelligence: A Strategic Enabler of National Security
    Open Source Intelligence (OSINT) has gained considerable prominence in recent years. Traditionally, intelligence has been the business of discovering secrets using a closed system of collection and analysis. Key sources included human intelligence (HUMINT), signals intelligence (SIGINT), and imagery intelligence (IMINT). Although open sources were frequently used in the intelligence process, their value was seen as secondary. Classified information was deemed more valuable and often more credible. The systematic acquisition of non-classified information was rarely seen as an intelligence priority.

    Today, OSINT’s importance is widely acknowledged. It is estimated that OSINT provides between 80 and 95 per cent of the information used by the intelligence community. There is a growing debate within and between the various branches of government and the national security apparatus on how best to use open source information. However, the role and potential of OSINT remain a matter of some dispute. OSINT’s advocates believe it to be the answer to many of today’s intelligence challenges. They call for a new intelligence paradigm marked by a preponderance of open source information and a trans-sector intelligence collaboration that includes a broad network of public and private actors. But there are others who warn against treating OSINT as more than a component of a continuing, all-source approach to intelligence-gathering and analysis.....

  13. #53
    Council Member
    Join Date
    Oct 2005
    Posts
    3,098

    Default

    House Committee on Homeland Security, Sep 08:

    Giving a Voice to Open Source Stakeholders: A Survey of State, Local and Tribal Law Enforcement
    ....Open source intelligence products can and should be shared with appropriate Federal, State, local and tribal law enforcement, and the private sector because of their unclassified nature. Unfortunately, DHS has not effectively exploited this type of information to provide essential analytical products. In fact, DHS’ efforts have lagged behind the rest of the Federal government. While the Office of the DNI and the CIA have each established robust open source programs, DHS – the lead Federal agency responsible for sharing terrorism threat and vulnerability information with State and local law enforcement – has yet to articulate a vision for how it will collect, analyze and disseminate open source information......

  14. #54
    Council Member
    Join Date
    Jan 2007
    Location
    Thunder Bay, Ontario, Canada
    Posts
    156

    Default DHS OSINT Sources for 2010 Olympics Security

    Mod Squad: If this belongs elsewhere, feel free to move w/my thanks for your patience.

    DHS has issued a Privacy Impact Assessment (PDF at DHS site - PDF at Scribd.com) highlighting some of the web sites they'll be monitoring to assess threats to the 2010 Olympics.

    In spite of my bitterness at not making it into the Appendix , I've developed a page with their selected list o' links here.

  15. #55
    Council Member AdamG's Avatar
    Join Date
    Dec 2005
    Location
    Hiding from the Dreaded Burrito Gang
    Posts
    2,548

    Default OPSEC, Open Source & the Black Budget

    An excellent example of how astute observers can take the innocuous and start piecing together puzzles.

    It is, according to a new book, part of the hidden reality behind the Pentagon’s classified, or “black,” budget that delivers billions of dollars to stealthy armies of high-tech warriors. The book offers a glimpse of this dark world through a revealing lens — patches — the kind worn on military uniforms.
    http://www.nytimes.com/2008/04/01/sc...prod=permalink

    Then again, when the media gets it wrong, they really get it wrong -

    http://trekmovie.com/2011/05/06/gema...laden-mission/
    A scrimmage in a Border Station
    A canter down some dark defile
    Two thousand pounds of education
    Drops to a ten-rupee jezail


    http://i.imgur.com/IPT1uLH.jpg

  16. #56
    Registered User
    Join Date
    May 2009
    Posts
    6

    Default

    I argue that when it comes to open source collection, the easiest target is the United States by a very wide margin.

  17. #57
    Council Member davidbfpo's Avatar
    Join Date
    Mar 2006
    Location
    UK
    Posts
    11,799

    Default Exploiting open sources: an example

    Leah Farrell's blog consistently provides insight, aided by being Australian too and from a police background:http://allthingscounterterrorism.com/

    A few weeks ago she posted a series of photos, obtained from a jihadi website, on an IMU training camp in Pakistan:http://allthingscounterterrorism.com...an-apparently/

    After some input and open source research she's posted an update:http://allthingscounterterrorism.com...ing-camp-pics/

    The IMU crop up irregularly, especially due to their German links and maybe there is nothing of note here. It is IMHO an example of what can be done from open sources.
    davidbfpo

  18. #58
    Council Member AdamG's Avatar
    Join Date
    Dec 2005
    Location
    Hiding from the Dreaded Burrito Gang
    Posts
    2,548

    Default

    For the Peanut Gallery

    If you have anyone in your chain who can take the Open Source course at Leavenworth, send them - it's well worth the time & effort.
    A scrimmage in a Border Station
    A canter down some dark defile
    Two thousand pounds of education
    Drops to a ten-rupee jezail


    http://i.imgur.com/IPT1uLH.jpg

  19. #59
    Council Member bourbon's Avatar
    Join Date
    Jun 2007
    Location
    Boston, MA
    Posts
    903

    Default OSINT: To Defeat Terrorists, Start Using the Library

    To Defeat Terrorists, Start Using the Library, by Scott Helfstein. Bloomberg, Aug 30, 2011.
    The information glut that marks the 21st century is evidenced in some unexpected places. Last month, my organization, the Combating Terrorism Center at West Point, released a report that sharply disputed conventional wisdom about terrorism along the Afghanistan-Pakistani frontier.

    The report argued that the Haqqani Network, a border- spanning tribal group with deep ties to Pakistan’s government, had been more influential than the Taliban in aiding al-Qaeda’s rise.

    How did we support this thesis, which has vast implications for reconciliation efforts in the region as well as for the distribution of U.S. military aid? Not with data culled from clandestine operations in Pakistan’s tribal areas or from Osama bin Laden’s computer hard drive. The report was based on the public statements and writings of individual extremists over the past 30 years. Rather than ferreting out secret information, researchers merely took extremists at their voluminous word.

    It seems terrorists, too, are susceptible to the syndrome known as Too Much Information.
    “[S]omething in his tone now reminded her of his explanations of asymmetric warfare, a topic in which he had a keen and abiding interest. She remembered him telling her how terrorism was almost exclusively about branding, but only slightly less so about the psychology of lotteries…” - Zero History, William Gibson

  20. #60
    Council Member
    Join Date
    Oct 2005
    Posts
    3,023

    Default

    bourbon, thanks for sharing this article. While in no way meaning to degrade or insult the excellent work our intelligent analysts do, intelligence analysis based on classified information (which is frequently just as questionable as unclassified information) often lacks critical context, especially historical context. And as in any other organization ideas tend to go viral and become accepted wisdom, even when they're incorrect. It is harder to correct the record when the idea is based on classified that not all the analysts (or subject matter experts) have access to.

    OSINT is an underused discipline in my view. I think part of the blame for our shortfalls is in our process, where commanders want dumbed down summaries on targets without context, so the intell community (not all of it, but definitely on the pointy end of the tactical spear) becomes target fixated, and they miss the bigger picture, a picture that may indicate their targeting process needs to be adjusted to have the desired effect.

Similar Threads

  1. Intelligence: failures, gaps and knowledge gaps
    By SWJED in forum Intelligence
    Replies: 5
    Last Post: 01-04-2017, 03:29 PM
  2. The David Kilcullen Collection (merged thread)
    By Fabius Maximus in forum Doctrine & TTPs
    Replies: 451
    Last Post: 03-31-2016, 03:23 PM
  3. Open Source Analysis of WikiLeaks?
    By BobKing in forum Intelligence
    Replies: 15
    Last Post: 09-11-2010, 04:08 PM
  4. Replies: 0
    Last Post: 11-25-2008, 10:28 PM
  5. Terrorist Groups Thesaurus / Open Source Guide
    By SWJED in forum Adversary / Threat
    Replies: 1
    Last Post: 11-17-2006, 03:19 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •