1. WHITE PAPER
BLACKBOX Multimodal Display Platform
Multimodal Interface Design as a
Solution for Data Display, Data
Fusion and Data Sharing
Author
Kurt Kratchman
kurt@blackbox.tv
Abstract – Today data comes at us from different directions in different formats,
structured and unstructured. Viewing so much data and creating data narratives
is challenging due to the various types of sources, supporting operating systems,
hardware, software, platforms, devices, channels, formats, codec, streaming
data, and proprietary data architectures, just to name a few. Fusing data is an
enormous challenge and opportunity we are focused on solving. The display
layer - the human computer interface - is the frontier this paper addresses. We
attempt to provide context, frame the problem statement and offer a strategic
paradigm and solution that is pragmatic, tactical and cost effective. The
functional role of a multimodal display platform is to assist the end user in
organizing and presenting multimedia content. At the lowest level, the display
platform organizes disparate data types and organizes relevant incoming data. At
the highest level, the system presents actionable intelligence to the user. In this
paper, we use the BLACKBOX model as a reference for displaying data driven
content, which utilizes multimedia fusion and multimodal characteristics to
enhance user performance. The design allows for the user to passively monitor
and actively interact with the display system with controls for interface
customization, template creation, interactivity with web services, high end design
and usability features, and a client / cloud based architecture. We present a new
display paradigm that utilizes a multimodal interface to (1) take advantage of the
human’s ability to fuse multisensory information, (2) facilitate situational
awareness, (3) enhance the users cognitive task analysis and (4) share
intuitively.
Keywords: adaptive interface, cognitive task analysis, command control, cross
platform, common operating picture, display bridge, display platform, force
multiplier, fusion, information sharing, multimodal interface design, multimedia,
situational awareness, synthetic vision, user interface, visual technology.

2. 1 Introduction
“We are drowning in data…”
The integration of senses by the human is a natural form of data fusion.
Extensions to human processing include computer displays to guide attention,
facilitate efficient task analysis, and provide cues for situation awareness. A
computer is good at processing large amounts of object data while a human is
good at making inferences over the data. Thus, the key question to ask is “What
framework the human has to inference over the data and what capabilities allow
for active reasoning?” To develop the user refinement capabilities to include in a
design, four issues are paramount (1) situational awareness, (2) cognitive task
analysis, (3) multimodal interface and (4) easy information sharing. The focus of
this paper is multimodal interface and describes a highly scalable display model
called BLACKBOX. BLACKBOX provides a multimedia interactive display
platform that transforms the way information is communicated. BLACKBOX is
specifically designed for fusion/command center analysts who need to view and
analyze data from secure and public sources in context. With BLACKBOX
dynamic data can be displayed by simultaneously force multiplying your existing
screens with video feeds, web services, analyst reports, digital content, charts
and graphs all on a single screen.
BLACKBOX makes having your own mission control affordable and easy to set
up. BLACKBOX gives IT and security organizations important enhancements
that not only help reduce support and remediation costs, but can also provide a
foundation for future solutions as security and communications needs change to
support increasingly virtualized or “multi-tenant” shared data center resources.
BLACKBOX is architected to be available as an externally or internally delivered
application that has built-in application and content syncing and caching. Users
can integrate a wide variety of content types and existing applications to be
delivered within BLACKBOX.
2 Multimodal Interface Design
Passive computer presentations might accurately present the fused data,
however if the mental model of the human operator is not consistent with the
display, the system usability might not improve performance. One way to improve
operator performance is to design an interactive interface that allows for the
capability to interact and refine the display. As an example, we might be able to
refine the data collection so as to get more image perspectives for a 3D target
presentation or multiple looks on the same target. When we interact with the
system, the end user is transformed from a passive operator to an active user.
Another way to improve human operator performance is to design a multimodal
interface. Multimodal interface design (MID) is a concept that leverages our
ability to filter incoming data. MID design is a key feature in military and
www.blackbox.tv 2

3. intelligence applications, security, aviation, simulators, weather analysis
platforms, and video games. MIDs augment display information with control
inputs, audio warnings, and some proprioceptive information. Such examples are
audio warnings for threat assessment and control interactions affording the
analyst the ability to cue in on data or drill down to areas of interest. Additionally,
we might think of classifying image fusion selections as an extension to the MID
design. For instance, allowing the user to select between electrical-optical (EO),
synthetic-aperture radar (SAR), and infrared (IR) for day, high-altitude, and night
missions, respectively, would enhance the multimodal visual capabilities. A key
component of the MID is the ability to facilitate the analyst’s cognitive ability to
fuse multiple image presentations for task completion.
Most fusion systems are highly specialized, designed as narrow point solutions,
and are expensive to build and operate, like the examples above. Some software
packages are so complex they are sold with their own analysts and engineers.
The typical Law Enforcement/intelligence fusion/command center cannot afford
such luxuries yet the operators are being asked to evolve their skills to handle
more time sensitive demands. What if the end user was empowered to select
their own data sets, live feeds, PowerPoint files, web services, maps and other
content for a fused display presentation with out support from software engineers
and consultants? What if they could set up templates of existing and approved
feeds and share them? This would assist in removing the need for time-
constrained resources and let the experts manage what they know best. More
times than not, the least dramatic fusions of data can have the greatest impact
on day-to-day productivity.
3 Content is Data, Rich Media, and Multimedia
As previously mentioned, the key to a multimodal interface system design is the
ability of the system to convey information desired by the user. With the growth of
rich media in all forms of communication and analysis, information technology
systems have more to deliver for users who handle and analyze multimedia.
Multimedia is media and content that uses a combination of different content
forms. The term can be used as a noun (a medium with multiple content forms)
or as an adjective describing a medium as having multiple content forms. The
term is used in contrast to media, which use only rudimentary computer display
such as text only, or traditional forms of printed or hand-produced material.
Multimedia includes a combination of text, audio, still images, animation, video,
or interactivity content forms. The key to consuming multimedia on display
screens is the interface, referred to as a GUI (graphical user interface).
Multimedia terminology has evolved over the years and conceptual terms such
as rich media, or layered media, have entered the professional lexicon.
More synthesized forms of content include the multimedia presentation of the
assets organized into PowerPoint, web pages and other curated forms of digital
content, often relied on during mission briefs and presentations. Digital content
can be stored locally or on remote servers such as cloud based data. The data
www.blackbox.tv 3

4. fragments fused can be web services and conventional application files such as
PowerPoint, web browsers, MP4 video assets and their players. (this seems
repetitive of the first sentence) Compatibility between operating systems, file
standards, proprietary software, and platforms make it difficult to fuse data into a
common view that can be presented on a display screen.
With our reliance of computer displays, interface design has become a critical
area of focus to increase performance. The objective of designers and
developers of interactive software systems is to align the mental model of the
human with the display of the data. One way UX (user experience) designers
improve operator performance is to design an interactive interface enabling the
human operator the capability to interact and refine the display.
Fig. 1 NASA Mission Control – Johnson Space Center
www.blackbox.tv 4

5. Fig. 2 NASA BLACKBOX
4 Force Multiply
Command centers and fusion centers are designed to provide situational
awareness and a common operating picture. Command centers such as NASA’s
are proliferating around the world. Sensors, surveillance and intelligence
solutions rely more and more on video imagery. Video displays are sharing the
real estate with computer screens. As prices go down, the demand for large
displays increases, driving new requirements and benefits. But most displays
are not utilized to their full potential. Historically, splitting the screens and running
multiple applications on a single view require additional hardware and software
components. Consumer oriented Smart TVs are leveraging web services and
web applications running through browser like application. While this makes the
viewing experience more dynamic, it is quite narrow and limited. In mission
critical command centers, rarely is budget spent on enhancing the usability of the
content that these architectures provide. Most system integrators and IT
professionals are not trained in user experience or interface design and such
resources have not been made a priority in the designers of NOCs or fusion
centers. Yet it is the human computer interface that is paramount to
understanding what the data means in context. Data presentation that creates
actionable intelligence saves lives and protects our security. Synthetic vision
supports our air force pilots. Where is the synthetic vision support for the analysts
who are piloting our security data centers?
www.blackbox.tv 5

6. 5 The Market for New Ways to Communicate
The market for new ways to communicate continues to grow and evolve
dramatically. Organizations expect and are willing to pay for, seamless
integration between their access to data, media and new technologies.
BLACKBOX consolidates data driven content through its unified digital display
platform.
Few organizations have the resources or expertise to manage large internal
teams of strategic, technical and creative personnel necessary to execute
comprehensive integrated digital media solutions in-house. BLACKBOX
empowers users to easily display multiple types of data and media tailored to
varying audiences for maximum impact. BLACKBOX’s award-winning interface
assembles new and existing technologies, enabling business and technical users
to combine information maintained in numerous databases across various
platforms. CIO’s and CFO’s are able to maximize their return on investment with
dozens of features that enables end users to access data that might not
otherwise be available or difficult to find.
6 Usage Models
MAJOR MILITARY DEFENSE CONTRACTOR. A leading military defense and
aerospace pioneer focused on intelligence, surveillance and reconnaissance
solutions for the US military worked with BLACKBOX to develop a customized
version of the BLACKBOX software as a multimodal interface paradigm for their
wide ranging software R&D and drone programs. The problem for most of these
companies is that they spend so much money on responding to RFPs, building
prototypes too slowly, developing intense hardware, software engineering and
system integration solutions that take months if not years of planning, design and
development, that by the time the end user operator gets to interface with the
system, they have run out of time or money to consider the adapting needs of the
end user and the context they need to see the data in. More often than not, the
needs of the ultimate end user are never studied or designed. Great ideas and
solutions die because of a failure to develop an interface that makes the system
usable. Hence interface design and usability together the next generation in
advancement for all software solutions and BLACKBOX is helping to set the
agenda.
NASA. Imagine the need to obtain approval for $30 million in funding; knowing
that the program you are trying to sell is extremely complicated and difficult for
even management to understand. This is the task that faces the engineers on
the Multi-Mission Space Exploration Vehicle Simulation program at NASA. The
team at NASA knows that in order for their project to survive they need be able to
demonstrate what they are attempting to accomplish with real data. Standard
presentation tools were not even an option. We proposed our solution to NASA
www.blackbox.tv 6

7. highlighting the depth and power of BLACKBOX including the ability to translate
a multitude of data points into graphs and charts (HDF5 conversion). To a
common viewer the data is meaningless, but to an engineer trying to analyze real
time and simulation data this is a valuable tool. The team at NASA recognized
the value in being able to show live satellite feeds, adjacent to sensor data views
in context with their plans and their mission – all as a means to get funding.
Project approved.
MAJOR EDUCATIONAL INSTITUTION. Private and public educators are forced
to adapt and augment curriculums, pedagogy, and content distribution to remain
relevant in a world of digital technology. A leading private content creator and
publisher has a news program broadcast via satellite to 8,000 middle and high
schools across the United States. The company even partners with the largest
broadcasting company in the world to help produce their content. Traditionally, all
of this content was broadcast on a single TV video screen in the classroom.
Recognizing the need to keep up with new technologies the company contacted
BLACKBOX to create an interactive learning platform. Desiring to engage
students and empower teachers even more, the educational leader will run a pilot
program utilizing BLACKBOX as a tool to augment content and provide a deeper
educational experience. BLACKBOX will be used to merge content and active
learning applications providing a more interactive approach to teaching and
learning. BLACKBOX offers schools a powerful tool to improve teacher
effectiveness. With the integration of interactive whiteboards, students truly
become active participants in the daily news broadcast, interacting with digital
content in real-time.
GLOBAL AUTOMOTIVE BRAND. This company is known throughout the world
for their innovations at every level in the global automotive industry.
Unfortunately marketing communication tools and messaging has not kept pace
with this rapidly changing market. A tailor-made version of BLACKBOX was
created for the marketing teams to be used as a sales tool to access information
about assets available to dealerships, as well as marketing resources that they
might not otherwise know existed. Management discovered that BLACKBOX is
also an effective tool for recruiting and employee on boarding.
7 How BLACKBOX Works
BLACKBOX provides the ability to display a multitude of media in a single
presentation screen: streaming video, web browsers, audio, RSS Feeds, Google
Maps, PDF, HDF5 data, graphs…the list of application driven “pods” goes on and
will continue to grow. There can be up to sixteen (16) pods on a screen. There
can be an unlimited number of screens, or “acts” that are the rotating sides of a
box. BLACKBOX allows this same content to be rearranged, overlaid or
enlarged on demand. The BLACKBOX platform allows multiple types of content
to be shared and distributed at the same time through an adaptive interface that
allows users to simultaneously control the display and organization of any type of
digital content.
www.blackbox.tv 7

8. 8 BLACKBOX Architecture
The BLACKBOX application runs as an externally or internally delivered
application that includes built in application syncing and caching. It is built using
Action Script 3 for the Adobe Air platform as a key technology. Adobe AIR is a
cross-platform runtime environment developed by Adobe Systems for building
Rich Internet Applications (RIA) that can be run as desktop applications or on
mobile devices. The Adobe AIR Runtime is the core of the application delivery.
Not unlike a JVM or other virtual machine, it provides a base layer for the
application.
BLACKBOX customers are provided with either an external or internal (behind
the firewall) installation of the system. Installations can be a single machine
running locally only or can consist of several machines with various
synchronization patterns as best suits the needs of the customer.
The BLACKBOX application is built on top of Adobe Air and provides core
features including playback, editing, content and application synchronization and
caching. The application is made up of several major components including:
• Authentication and Authorization. The application validates a single
installation on a single machine and ensures appropriate authorization
to access the installer.
• Update Manager. This is used to keep application versions consistent
within an installation. Update manager also can push content as part of
a regular update.
• Playback. The application is able to execute playback of a series of
acts (pages) and pods (individual windows). Pods can consist of a
wide variety of different media types and applications and can be
expanded.
• Content Editor. This allows users update the content that will be
played via each instance of the application. Through the update
manager this will be pushed across the installations.
www.blackbox.tv 8

9. The BLACKBOX application development follows best practices established by
Adobe Technical Services and partners. We use Cairngorm as the foundation on
how we deliver successful enterprise Flex and AIR projects. Most large-scale
applications within Adobe Technical Services are built by applying patterns and
practices of Cairngorm 3 using Inversion-of-Control (IoC) containers. Design
patterns advocated with Cairngorm MVC, such as encapsulating operations
within command objects, declaring models and services externally, and
separating presentation and domain concerns, can be applied with Inversion-of-
Control containers to simplify development and testability. The following diagram
illustrates the BLACKBOX software framework.
Fig. 3 BLACKBOX Framework Diagram
www.blackbox.tv 9

10. A Cairngorm application consists of:
• Architectural Layers that separate classes with different kinds of
responsibility
• Functional Areas that group classes related to the same area of
functionality
• Design Patterns that coordinate objects in consistent ways
Architectural Layers
Classes with different kinds of responsibility belong to different layers than can
be changed separately from one another. So if the visual design needs to be
tweaked, most changes are confined to the classes in a presentation layer, and
not other parts of the code. Cairngorm recommends the layers described by Eric
Evans in Domain-Driven Design:
1. Presentation - presents data to the user and gathers input. i.e. the fancy
UI
2. Application - performs the operations of the application. i.e. saving a form
3. Domain - models the business concerns of the application. i.e. the form
data
4. Infrastructure - coordinates objects and integrates with other systems. i.e.
talking to the server
Extensions
The BLACKBOX architecture uses an extend relationship to specify that one use
case (extension) extends the behavior of another use case (base), increasing
product flexibility and adaptability. This type of relationship reveals details about
a system or application that are typically hidden in a use case. While the base
use case is defined independently and is meaningful by itself, the extension use
case is not meaningful on its own. The extension use case consists of one or
several behavior sequences (segments) that describe additional behavior that
can incrementally augment the behavior of the base use case. Each segment
can be inserted into the base use case at a different point, called an extension
point. The extension use case can access and modify the attributes of the base
use case; however, the base use case is not aware of the extension use case
and, therefore, cannot access or modify the attributes and operations of the
extension use case. The diagram below illustrates the BLACKBOX extensions.
www.blackbox.tv 10

11. Fig. 4 BLACKBOX Extension Diagram
Component Model
BLACKBOX utilizes the Spark component architecture to support a seamless
way to customize the visuals and behaviors of components in a much more direct
and effortless manner. Spark is the name of the new component skinning
architecture and component set that is part of the Adobe Flex 4 open source
software development framework. The main tenet of the Spark architecture is a
skinning model that provides a neat separation of a component’s visual elements
from its logic. This separation gives designers and developers more freedom,
because the visual elements of a Flex component can be designed in an
unfettered manner, independent of the implementation of the logic powering the
component. The following diagram illustrates the BLACKBOX Spark Component
Model.
www.blackbox.tv 11

12. Fig. 5 BLACKBOX Component Model
Class Diagram
BLACKBOX class diagram in the Unified Modeling Language (UML) describes
the structure of the system by showing the system's classes, their attributes,
operations (or methods), and the relationships among the classes. The class
diagram is the main building block of object-oriented modeling. It is used both for
general conceptual modeling of the systematics of the application, and for
detailed modeling translating the models into programming code. The classes in
the class diagram below represent both the main objects and or interactions in
the application and the objects to be programmed. In the BLACKBOX class
diagram below these classes are represented with boxes which contain two
parts:
• The upper part holds the name of the class
• The bottom part gives the methods or operations the class can take or
undertake
www.blackbox.tv 12

13. Fig. 6 BLACKBOX Class Diagram
9 API
The BLACKBOX API provides a way to extend the core capabilities
of BLACKBOX in a seamless way. The Open Pod provides a container that is
similar to modern web browsers. HTML, CSS and JavaScript are used to
present content and access BLACKBOX functionality through JSON. The API
www.blackbox.tv 13

14. includes the ability to access the raw HTML DOM. Taken together this provides
rich hooks to content objects and functionality giving a simple mechanism that is
widely used by software engineers as a means to build interesting extensions
BLACKBOX.
10 Security
When discussing security for BLACKBOX applications, it's first important to
understand that this is a controlled delivery environment that is quite different
from the security issues that exist when delivering Flash or Adobe AIR
applications across public web sites. Most installations use the controlled
environment as the first layer of protection that eliminates entry of malicious
code. Coupling that with code signing from Adobe AIR provides a high degree of
protection. Adobe AIR requires developers to digitally sign their applications in
order to maintain the integrity of the software and the identity of its publisher.
BLACKBOX has a certificate from one of the most respected Certification
Authorities (CA) - Thawte. Digitally signing AIR files provides significant
assurance to users that the application they are installing has not been altered.
That said, BLACKBOX is being built to work in public installations that will have
broader security concerns. Thus, we've architected it to leverage several of the
core security features of the Adobe AIR framework:
Sandbox
The Adobe AIR runtime like other kinds of JVMs prevents code from running
directly on the machine. Further, the Adobe AIR sandbox provides additional
security that limits application access according to given security specifications.
BLACKBOX leverages the sandbox to restrict access of the application.
Memory Restrictions
BLACKBOX also leverages the built in memory restrictions of Adobe AIR that
prevent buffer overflow, memory corruption or other cross application memory
issues.
Data Encryption
For added security we have initiated the Data Encryption to store local data in an
AES-CBC128-bit encryption. This ensures that local data cannot be attacked or
corrupted by malicious attacks. Further we restrict local data to non-script data,
i.e., that is not executed. Rather it is only content oriented information that would
cause playback issues but is not an entry for execution.
HTML Security
BLACKBOX is capable of rendering HTML that includes untrusted JavaScript.
This is greatly restricted by the sandbox and the AIR runtime.
References
www.blackbox.tv 14

15. Further reading around Adobe AIR security can be found in the following
references:
• http://help.adobe.com/en_US/air/security/air_security.pdf
• https://docs.google.com/open?id=15SBR17kOVz8kVADOBgIGqAacKe
jhNSCKSEMNf0J0igbtV-Bc3yJJy_DbGCYy
• http://help.adobe.com/en_US/air/security/index.html
11 BLACKBOX System Requirements
The system requirements for BLACKBOX playback are as follows:
Windows
• Intel® Pentium® III 1GHz or faster processor; Intel Pentium 4 2GHz or
faster
• Microsoft® Windows® XP Home, Professional, or Tablet PC Edition with
Service Pack 2 or 3 (including 64-bit editions); Windows Server® 2003 or
2008; Windows Vista® Home Premium, Business, Ultimate, or Enterprise
(including 64-bit editions) with Service Pack 2; or Windows 7 (including
64-bit editions)
• 512MB of RAM (1GB recommended)
MAC OS
• Intel Core™ Duo 1.83GHz or faster processor
• Mac OS X v10.5 or v10.6
• 512MB of RAM (1GB recommended)
12 SUMMARY
Most enterprise software tools execute only on functional requirements and
system architectures designed in isolation of other point solutions,
communication systems and end user contexts. The demands for integration and
sharing across channels are more dynamic than ever. Today’s businesses and
organizations need this kind of enhanced productivity to help optimize critical
customer, employee, and financial data, and preserve systems infrastructure.
This grows ever more crucial as companies adopt more virtualized, shared, and
multi-tenant infrastructure models.
With BLACKBOX enabled solutions you can:
• Leverage existing infrastructure and force multiply the views of data to
help make informed decisions
• Improve productivity and operator effectiveness by fusing disparate data
feeds into a multimodal interface
• Improve communications across the enterprise, with customers and
constituents
www.blackbox.tv 15

16. • Give (consistency in tense) control to the analyst to customize their data
views without needing support from designers, front-end coders, and IT for
rapid data fusion - reduce strain on IT resources and limited budgets
• Share complex data views at multiple levels on multiple platforms
• Address the increasing and evolving security threats across your physical
and virtual infrastructure through display enhancements
• Facilitate compliance with government and industry regulations and data
protection standards
• Reduce precious real estate costs by using less hardware
• Reduce hardware costs by reducing the need for more displays and
dedicated computers - doing more with less
• Reduce user fatigue by removing all the visual noise from the useful
content the operator needs to focus on
“Few organizations have the resources or
expertise to manage large internal teams of
strategic, technical and creative personnel
necessary to execute comprehensive integrated
digital media solutions in-house.”
Overall, BLACKBOX is enabling a significant opportunity for organizations to
“future proof” their infrastructures. Using BLACKBOX-enabled solutions, they
can get ahead of the curve of emerging technologies. IT organizations can gain
important data display solutions for their growing virtualized environments to
allow them to better control the flow of confidential, privileged, or sensitive
workloads or data by restricting these to more thoroughly evaluated or trusted
platforms. While at the same time, enable their customers to rapidly exploit new
data feeds with minimum strain on existing infrastructure. While the near-term
model will be the creation of up-sell and cross sell strategies amid their legacy
system vendors, increasingly, BLACKBOX platform trust will grow on the ability
to take advantage of the data to create unfair advantage for the “good guys.”
Having the BLACKBOX platform enables a baseline level of assurance as
systems are refreshed - essentially “raising the bar” for fusion and data center
optimization, profitability and security over time.
13 Suggested Reading
1. EDWARD TUFTE, Visual Explanations: Images and Quantities,
Evidence and Narrative, Graphics Press (February 1997), ISBN-10:
0961392126.
www.blackbox.tv 16

17. 2. EDWARD TUFTE, The Visual Display of Quantitative Information,
Graphics Press; 2nd edition (May 2001), ISBN-10: 0961392142.
3. EDWARD TUFTE, Beautiful Evidence, Graphics Pr; 1St Edition (July
2006), ISBN-10: 0961392177.
4. STEPHEN FEW, Information Dashboard Design: The Effective Visual
Communication of Data, O'Reilly Media; 1 edition (January 1, 2006),
ISBN-10: 0596100167.
5. JERALD HUGHES, Baruch College, CUNY and KARL LANG, Baruch
College, CUNY, White Paper: “Transmutability: Digital
Decontextualization, Manipulation, and Recontextualization as a New
Source of Value in the Production and Consumption of Culture
Products,” Proceedings of the 39th Hawaii International Conference on
System Sciences, 2006.
6. DIMITROS TZOVARAS, Multimodal User Interfaces: From Signals to
Interaction (Signals and Communication Technology), Springer;
(November 25, 2010), ISBN-10: 3642097030.
7. OLIVIERO STOCK (Editor), MASSIMO ZANCANARO (Editor),
Multimodal Intelligent Information Presentation (Text, Speech and
Language Technology), Springer; 1st edition (April 29, 2005), ISBN-10:
1402030495.
8. ANDREW LOSOWSKY, Visual Storytelling: Inspiring A New Visual
Language, Gestalten, Berlin, 2011, ISBN 978-3-89955-375-8.
www.blackbox.tv 17