Although open-source intelligence has been utilized by the intelligence community for more than 50 years, analysts have had to adapt to ever-advancing technology and social platforms. With the creation of Facebook, YouTube, and Twitter in 2005 and 2006 has come a new wave and definition of open-source intelligence. In the RAND Corporation’s Defining Second Generation Open Source Intelligence (OSINT) for the Defense Enterprise, Heather J. Williams and Ilana Blum break down open-source intelligence capabilities into three generations, including its beginnings during World War II, where it is now after the rise of the internet and social media, and where it is shifting to with the increase in encryption and advances in machine learning.
During the early days of open-source intelligence, open source information was more limited and harder to gain access to in comparison to the limitless online forums we have access to today. The first open-source intelligence service created in 1941, the Foreign Broadcast Monitoring Service (FBMS), was mainly tasked with monitoring and analyzing the Axis powers’ propaganda. Thus, the early stages of open-source intelligence focused on translation from foreign languages, collection, analysis, and dissemination. It required physical access to sources and was limited by regular publication cycles. The open-sources were largely constrained to radio broadcasts, newspapers, journals, books, documents, and other tangible pieces.
Though the first wave of open-source intelligence faced its fair share of source and methodology limitations, open-source analysts’ work during the Cold War provided critical insight and decisions for the military. It is estimated that 80% of information used to monitor the collapse of the Soviet Union was pulled from open sources.
After a period of intelligence community budget following the end of the Cold War, leaders began to recognize that the technological shifts and increasing connectivity occurring with the rise of the 21st century would facilitate a higher demand for open-source intelligence. Deputy Director J. Niles Riddel of Federal Broadcast Information Service (FBIS), the open-source intelligence service of the time, attributes the shift to the rise in “personal computing, large-capacity digital storage, capable search engines, and broadband communication networks.” This recognition and an inherent underfunding of open-source intelligence capabilities facilitated the creation of the Defense Open Source Council (DOSC) for the Department of Defense in 2004 and the Open Source Center (OSC) for the Central Intelligence Agency in 2005.
Today, open-source intelligence is defined by the RAND Corporation as “publicly available information that has been discovered, determined to be of intelligence value, and disseminated by a member of the IC.” Open-source intelligence analysts work with open-source information (OSIF), which includes unclassified data available to the public and data that can be obtained through request or observation. Individually, this data would be of little value, but collectively it can lead to important insights. OSIF can come from news media content, gray literature, which includes content from non-media institutions and organizations, long-form social media content such as data from blogs and sites such as Reddit and Tumblr, and short-form social media content such as content from platforms such as Facebook, Twitter, and LinkedIn.
The open-source intelligence cycle contains four key steps: collection, processing, exploitation, and production. Further, each step includes sub-sets: collection includes acquisition and retention, processing includes translation and aggregation, exploitation includes authentication and evaluating credibility, and production includes classification and dissemination. In today’s media-oriented social sphere, the intelligence community must focus on the management of big data, be wary of legal issues related to privacy rights, and acknowledge the skewed social media aggregation towards United States users.
In terms of processing, the second-wave of open-source intelligence has become easier with new analytical tools and methods, including Google Translate. But like the collection step, the processing step faces new challenges in terms of processing big data in less-structured formats. The abundance of data complicates the exploitation phase, but the modern access to news articles and online-published gray literature allows for fairly simple authentication. Further, the production phase today is able to make use of many different mediums, including written reports, briefings, videos, audio, and interactive graphics.
The open-source intelligence analysts today also have access to a myriad of different analytical tools that did not exist in the early stages of intelligence. For example, the intelligence community today can utilize social media analysis methods such as lexical analysis and social network analysis, as well as supplement their findings with geospatial context to create meaningful intelligence. For instance, the intelligence community has been able to utilize geotagged social media posts to track down Islamic State fighters. Thus, open-source intelligence today is focused primarily on exploitation and production,
It is of value to recognize where the web is going, both on the user end and the analytical end. What technology experts deem the “Semantic Web”, the third wave of open-source intelligence will have to possess the capabilities for “direct and indirect machine processing of data, machine learning, and automated reasoning.” Thus, the third wave of open-source intelligence will likely be built on machine learning and automated reasoning, have to deal with pervasive encryption denying access to OSIF, and may focus on collection and dissemination of big data.
Our analysts here at Praescient Analytics have lived these changes. As dedicated intelligence professionals, it is our mission to adapt to shifts in the open source environment. We support our clients by remaining at the forefront of innovation, partnering with up and coming technology providers and tracking industry trends. As we enter the third wave of open-source intelligence, our analysts are ready for the challenge.