If there’s one point on which national security planners and international investors would agree, it’s that the decisions surrounding the commitment of resources abroad have become increasingly complex. The potential for rapid shifts in political, security, and social dynamics the world over—the events of the Arab Spring remain one of the best and most frequently cited examples of unexpected upheaval, but the recent protests in Turkey and Brazil are also noteworthy—have injected international operations with significant risk. And yet between the massive proliferation of new forms data on the web and the emergence of powerful analytic platforms to parse and understand this open source information, there has never been a more exciting time to be an analyst examining risks and opportunities in frontier environments.

Analysts today have a wealth of rich data at their disposal, whether in the form of real-time news and social media feeds (which can be helpfully aggregated, structured, and visualized with tools from companies like Recorded Future), or the vast collection of economic, financial, demographic, and health datasets freely available, for example, at Quandl. The formal release earlier this spring of a huge dataset of carefully selected, geo-referenced political and security events dating back to 1979 and updated daily, known as the Global Database on Event, Location, and Tone (GDELT) led to excited speculation among policy analysts about its likely utility and impact. In short, with such easy access to a wide variety of data, it’s possible for savvy analysts equipped with the right technologies to observe, understand, and develop assessments of the facts on the ground in any given country or region in near real-time—while at the same time placing trends in historic context.

The value of open source data and the analysis it enables has become particularly apparent in the national security community. The wars of the last decade have demonstrated the way in which intelligence analysts, military planners, and policy makers all require a shared, granular, understanding of the people, politics, and economics of the countries in which the U.S. and its allies are engaged. Few traditional intelligence collection and analysis capabilities can create as comprehensive and collaborative an analytical picture as those rooted in open source data. In the years ahead, we expect that reliance on open source data and analysis will become more prominent particularly in those environments where U.S. and allied government assets are absent or constrained. 

The ongoing crisis in Syria has demonstrated the way in which data culled online from social media can produce otherwise unattainable insights. In 2012, analysts from the Institute for the Study of War, one of Praescient’s partners in the think tank community, were able to thoroughly map the structure and composition of the Free Syrian Army using information gathered from the many videos released on YouTube by groups of fighters as they joined the rebellion. ISW and another of Praescient’s partners, the Institute for the Study of Violent Groups (ISVG), have successfully informed policy debates and driven military operational decisions by producing volumes of granular analysis and sharing datasets developed through a process of careful open source collection.

Many of the advantages that open source analysis presents in the national security community can be carried over directly to the private sector, where data-driven decision-making is now the standard across industries. Investments and operational planning in emerging market environments, in particular, can be greatly informed by the rapid analysis of real-time data available online about political, security, and social developments in countries and regions of interest. In addition, there are a growing number of firms exploring the possibility of integrating social media sentiment monitoring into their analysis of financial markets. And like analysts in the national security community, many of whom now serve effectively as “curators” of data—swiftly collecting, organizing, and validating new and emerging forms of open source information, then integrating that data with more traditional classified materials—risk analysts in the private sector will also generate new insights by fusing purposefully collected open source intelligence with their companies’ internal planning data.

So, in the end, what’s the principal lesson about open source analysis for the national security and international business communities? In short: today’s open source data, combined with powerful analytic technologies (like Palantir, for example), leads to better analysis, better decisions, and better outcomes—all delivered more quickly than at any point in the past. Praescient, for its part, is pleased to be leading the development of a community of analysts who are trained and equipped to serve as the new curators of data, helping clients achieve rapid insights and impact that only web intelligence can enable.