Managing data in motion—the flow of information through and between different systems—is mission-critical for both the intelligence community and virtually every commercial enterprise.
Technology developed by the National Security Agency (NSA) to route and prioritize intelligence information for mission use has been released to the open-source software community and adapted to serve the growing big data needs of industries ranging from energy and communications to the emerging Internet of Things (IoT).
To ensure that the intelligence community (IC) gets the most valuable information to those who need it first, the NSA developed Niagarafiles, technology that automatically manages the flow of data across systems. Niagarafiles automates the management, manipulation, and storage of large streams of data in real time. The technology can understand and transform data in a variety of formats, easing transfer across different systems and technologies. To prioritize data flows.
Niagarafiles makes complex routing decisions based on both the content of the data and its context. The technology also provides interactive command and control of data flows, allowing engineers to quickly change how data is assessed and transmitted.
The technology can understand and transform data in a variety of formats, easing transfer across different systems and technologies. To prioritize data flows.
By automatically embedding context into discrete data flows as they move across and between systems, Niagarafiles creates a fine-grained chain of custody for information, which led to organic interest across the IC.
First created in 2006, Niagarafiles was rapidly adopted for mission use by the IC over the next three to four years. In 2014, the technology was released to the open-source community as Apache NiFi, and more than 150 unique contributors have made improvements that have benefitted both the NSA and its commercial users. By releasing its source code to the open-source software community, the NSA both increased NiFi’s agility and security, and brought to commercial markets the big data technology now used by hundreds of companies worldwide.
Fortune 500 companies ranging from ExxonMobil and Ford to AT&T and Lenovo are now using open-source and commercial adaptations of NiFi to manage data flows. For the NSA, open sourcing the technology has led to significant improvements relevant to its mission as well as identified vulnerabilities that could be corrected to strengthen the tool’s capabilities. The NSA technology transfer program also used NiFi’s open sourcing process to standardize its procedures for the future release of other technologies, resulting in more than 30 additional technologies being released into the public domain, with benefits for both the Agency and the nation’s economy.
Contact: Linda Burger, (443) 634-3518, [email protected]