Semantic, Cognitive and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing

Amit Sheth
Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing:
Wright State University, Dayton, Ohio
Semantic, Cognitive, and Perceptual Computing:
three intertwined strands of a golden braid of intelligent computing
Keynote @ International Summer School on Web Science & Technology (WebST), Bilbao, Spain, 19 May 2016.

http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7433358

Credit: Looi Consulting (http://www.looiconsulting.com/home/enterprise-big-data/)
● In 2008, data generated > storage available. Less than 0.5% of data get analyzed.
● Vast variety of data: text > images > A/V > genome sequencing > IoT
● Of all the data generated, which data is relevant, and why? Which data to analyze? Which
data can offer insight? Who cares for what data? How to get attention to a human decision
maker? What we need is intelligent processing to get actionable, smart data.
A Big Challenge and Opportunity in Recent Times

How would an enterprise get actionable information?
http://www.slideshare.net/NamrataChatterjee/nokias-supply-chain-management-case-study, http://www.economist.com/node/7032258
● Weak crisis judgement.
● Failure to take prompt action.
● Single supplier reliability.
Fire at Royal Philips electronic
semiconductor plant, New
Mexico in March 2000.
8 trays of wafers
containing the
miniature circuitry to
make several
thousand chips for
mobile phones was
destroyed.
The expected time to recover was estimated to be a week.
● Fire breakout in clean room.
● Inability to determine the exact
damage to the clean room.
● Lack of emergency preparation.
● Early speculation of possible crisis.
● Preparedness against supply crisis.
● Finding alternative source of chip
supply.

http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/
The Patient of the Future
MIT Technology Review, 2012
How would an individual get actionable information?

First used in 2004 redefined 2013: http://wiki.knoesis.org/index.php/Smart_Data.
Smart Data

● The astounding bandwidth of your senses is 11
million bits of information every second.
● In conscious activities like reading, the human
brain distills approximately 40 bits of
information per second.
http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.html
The Brain: Inspiration for Intelligent Processing:
What if we could automate such interpretation of data?

● How can we take inspiration from human brain and derive an intelligent processing of big data
○ for service enterprise need to serving individual needs
● In particular, take inspiration from cognition and perception, and for pedagogy, outline three
computing paradigms for building future intelligent systems focused on converting data into
abstractions that humans act upon for making decision and taking action
○ not all aspects of capturing human intelligence are addressed
○ not focused on making machines that replace humans - instead focus is on CHE
● CHE: Humans and machines partner to enhance human experience and better informed decision
making with humans in the Loop.
* 2010 article: http://wiki.knoesis.org/index.php/Computing_For_Human_Experience; distinct from “Human Experience of Computing by Booch:
Dec. 9, 2011 at http://computingthehumanexperience.com/category/computing/.
Sheth - 2008+ *
What is this talk about?

http://www.livescience.com/1863-theory-intelligence-works.html, http://www.wired.com/2015/10/scientists-can-now-predict-intelligence-brain-activity/
Human Intelligence is not confined to
a single area in the brain, but is the
result of multiple brain areas working
in coordination.
Human Intelligence is related to how
information travels through the brain.
According to a review of 37 imaging studies in Journal Behavioral and Brain
Science, intelligence is related not so much to brain size or brain structure,
but to how efficiently information travels through the brain.
Some brains are better than others at
certain things because of the way they are
wired.
“The more certain regions are talking to
one another, the better you are able to
process information quickly and make
inference.”
Intelligence Research

Brain
Neuroscience
Cognitive
Science
Linguistics
Smart Data and
Intelligent Computing
● Semantic
Computing
● Cognitive
Computing
● Perceptual
Computing
Cognitive
Psychology
….
….
Brain Inspiration

http://www.research.ibm.com/cognitive-computing/brainpower/, https://newtonsapplevce.wordpress.com/2016/01/06/neuromorphic-technology/
Recreating the brain in silicon
One Approach to Intelligence

A Second Approach to Intelligence

Prior Medical Knowledge
D1
D3
D1
Medical History and
Past observations
S1
S2
S3
Sn
..
..
D1
D2
D3
S1
S2
S3
..
..
..
..
..
..
..
..
D1
D1
Doctor Patient
Q1
Q2
Qn
A1
A2
An
Blood Pressure
Heart Rate
Breathing Rate
Body Temperature
Multi-model Observations
Current: Observing a Snapshot of the Patient

ACTIONS
situation awareness useful
for decision making
ABSTRACTIONS
make sense to humans
KNOWLEDGE
for interpretation of observations
Contextualization
Personalization
DATA
Observations from machine
and social sensors
Converting Data to Actions

1Marcus, Philip, Kevin R. Murphy, Abid Rahman, and Christopher D. O’Brien. "Intrapatient symptom variability in
adults and children with asthma: Results of a survey." Advances in therapy 22, no. 5 (2005): 488-497.
“ … survey indicates that adult patients and caregivers of pediatric patients
report variability in asthma symptoms over time, even when asthma medications are taken.”1
Personal level
Signals
Public level
Signals
Population level
Signals
Future: Analyzing a Multifaceted Continuous Stream of Diverse Data

How do we solve problems with real-world complexity, gather vast amounts of data,
diverse knowledge, and come up with intelligent decisions and timely actions?
Next, a pedagogical approach.

Semantics, perception, and cognition interact seamlessly.
● Semantic Computing can deal with big data challenges.
● Cognitive Computing can use relevant knowledge to improve data
understanding for decision-making.
● Perceptual Computing can provide personalized and contextual abstractions
over massive amounts of multimodal data from the physical, cyber, and social
realms.
https://www.linkedin.com/pulse/perceptual-computing-third-strand-golden-braid-amit-sheth
Semantic Computing, Perceptual Computing, Cognitive Computing

http://www.mezzacotta.net/garfield/?comic=1470, https://in.pinterest.com/pin/222435669066482336/
● Semantics is the “meaning or
relationship of meanings, or relating to
meaning ” (Webster).
● Meaning and use of data
(Information System).
● It is concerned with the relationship
between the linguistic symbols and
their meaning or real-world objects.
● Semantics important role is to make
data meaningful to people.
Semantics

Semantics attaches meaning to observation by providing a definition within a system context
or the knowledge that people possess [Sheth 2016].
Semantic Computing encompasses the technology required to represent concepts and their
relationships in an integrated semantic network that loosely mimics the brain’s conceptual
interrelationships.
Web of data
Semantics attached to
objects in the world
Semantic Computing

Population Level
Personal
Wheeze – Yes
Do you have tightness of chest? –Yes
ObservationsPhysical-Cyber-Social System Health Signal Extraction Health Signal Understanding
<Wheezing=Yes, time, location>
<ChectTightness=Yes, time, location>
<PollenLevel=Medium, time, location>
<Pollution=Yes, time, location>
<Activity=High, time, location>
Wheezing
ChectTightness
PollenLevel
Pollution
Activity
Wheezing
ChectTightness
PollenLevel
Pollution
Activity
RiskCategory
<PollenLevel, ChectTightness, Pollution,
Activity, Wheezing, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
.
.
.
Expert
Knowledge
Background
Knowledge
tweet reporting pollution level
and asthma attacks
Acceleration readings from
on-phone sensors
Sensor and personal
observations Signals from personal,
personal spaces, and
community spaces
Risk Category assigned by
doctors
Qualify
Quantify
Enrich
Outdoor pollen and pollution
Public Health
Well Controlled - continue
Not Well Controlled – contact nurse
Poor Controlled – contact doctor SSN
Semantic Computing for Our Personalized Digital Health (Asthma) Application

Cognition is the process by which an autonomous system perceives its
environment, learns from experiences, anticipates the outcome of the events,
acts to pursue goals and adapt to the changing environment.
https://mitpress.mit.edu/books/artificial-cognitive-systems
PerceptionAction
Anticipate
Adapt Assimilate
Cognition as a cycle of anticipation, assimilation,
adaptation: embedded in, contributing to, and benefitting
from a continuous process of action and perception.
What is Cognition?

DARPA launched a Cognitive Computing project in 2002. Cognitive Computing was defined as
the ability to “reason, use represented knowledge, learn from experience, accumulate
knowledge, explain itself, accept direction, be aware of its own behavior and capabilities,
[and] respond in a robust manner to surprises.”
IBM describes the components used to develop, and behaviors resulting from, “systems that
learn at scale, reason with purpose and interact with humans naturally.” According to them,
while sharing many attributes with the field of artificial intelligence, it differentiates itself via
the complex interplay of disparate components, each of which comprise their own individual
mature disciplines [1].
Our Take
Cognitive computing interprets annotated observations obtained from Semantic computing ,
or raw observations from diverse sources and presents actionable information to humans.
Cognitive computing systems learns from their experiences and improve when performing
repeated tasks.
[1] https://en.wikipedia.org/wiki/Cognitive_computing
Cognitive Computing

http://www.rogerschank.com/fraudulent-claims-made-by-IBM-about-Watson-and-AI
Need to be Cautious with Claims (Roger Schank)

“where we see the world we do not decide to see it” Daniel Kahneman
Perception is an active cyclical process of exploration and
interpretation.
Perception enables individual to focus on most promising
course of action by incorporating background knowledge that
provided a comprehensive contextual understanding
Perception

http://www.ted.com/talks/blaise_aguera_y_arcas_how_computers_are_learning_to_be_creative#t-345661
Image of a bird and a computer says it’s a bird.
Perception: Google

Perceptual computing is the ability for a computer to recognize what is going on around it.
Computer can perceive the environment and the users in that environment. The computer
determines what needs a user might have and react to those needs without giving or
receiving any additional information.
Perception: Intel

http://www.jfsowa.com/pubs/cogcycle.pdf, http://www.jfsowa.com/talks/interop.pdf
Peirce’s Cycle of Pragmatism
John Sowa’s Perception Cycle

Perceptual Computing supports the ability to:
● Ask contextually relevant and personalized questions.
● Complement Semantic Computing and Cognitive Computing by providing the machinery to
ask the next question or derive a hypothesis.
It uses machine perception together with available background knowledge to explore and
interpret observations.
Perceptual Computing system enables the personalization of information provided by the
cognitive computing system.
For example, it minimizes the uncertainty by providing a personalized and contextualized
understanding of a patient’s environment and symptomatic variations.
Perceptual Computing: Our View

Hyperthyroidism
Elevated
Blood
Pressure
Systolic blood pressure of 150 mmHg
“15
0”
... ...
Levels of Abstraction

Making Sense of Sensor Data With
Henson, et al. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web, Applied Ontology, 2011.
Our First Approach to Implement Perceptual Computing

Explanation is the act of choosing the objects or events that best account for a set of
observations; often referred to as hypothesis building
Discrimination is the act of finding those properties that, if observed, would help distinguish
between multiple explanatory features
1
2
Explanation and Discrimination

Explanatory Feature: a feature that explains the set of observed properties
elevated blood pressure Hypertension
Hyperthyroidism
Pulmonary Edema
Clammy Skin
Palpitations
Explanation

Discriminating Property: is neither expected nor not-applicable
clammy skin
Hypertension
Hyperthyroidism
Pulmonary EdemaPalpitations
elevated blood pressure
Discrimination

Orders of magnitude resource savings for generating and storing relevant abstractions vs. raw observations.
Relevant Abstractions
Raw Observations
Resource Savings of Abstracting Sensor Data

Providing actionable information in a timely manner is
crucial to avoid information overload or fatigue
Sleep data Community dataPersonal
Schedule
Activity data Personal health
records
Data Overload for Patients/Health Aficionados

Personal level
Signals
Public level
Signals
Population level
Signals
Domain
Knowledge
http://www.tuberktoraks.org/managete/fu_folder/2011-03/html/2011-3-291-311.html
Contextual ActionablePersonalized
OR
How is my Asthma control?
Should I take additional medication today?
How can I reduce my asthma attacks at home?
Asthma: Challenges in Heterogeneity, Variability, and Personalization

Sensordrone
(Carbon monoxide,
temperature, humidity)
Sensor Platforms
Android Device
(w/ kHealth
App)
Total cost: ~ $550
Along with sensor platforms in the kit, the application uses a variety of population
level signals from the web:
Pollen level Air Quality Temperature & Humidity
Node Sensor
(exhaled Nitric Oxide)
Fitbit ChargeHR
(Activity, sleep quality)
kHealth Kit for the Application for Asthma Management

For collecting observations from both machine sensors
and from patients in the form of a questionnaire.
kHealth Kit: Android Application

kHealth Kit: Android Application

Personal level
Signals
Public level
Signals
Population level
Signals
Domain
Knowledge
Risk Model
Events from
social streams
Take medication before
going to work.
Avoid going out in the
evening due to high pollen
levels.
Contact a doctor.
Analysis
Personalized
Actionable
Information
Data Acquisition &
aggregation
kHealth: Health Signal Processing Architecture

Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
GREEN – Well controlled
YELLOW – Not well controlled
Red – Poor controlled
How controlled is my asthma?
Patient Health Score (Diagnostic)

Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
Patient health
Score
How vulnerable* is my control level today?
*considering changing environmental conditions and current control level
Patient Health Score (Prognostic)

Sensordrone – for monitoring
environmental air quality
Wheezometer – for monitoring
wheezing sounds
Can I reduce my asthma attacks at night?
What are the triggers? What is the wheezing level?
What is the propensity toward asthma?
What is the exposure level over a day?
Commute to work
Luminosity
CO Level
CO in gush during
the daytime.
Actionable
Information
Personal level
Signals
Public level
Signals
Population level
Signals
What is the air quality indoors?
Close the window at home during day to avoid
CO2 inflow, to avoid asthma attacks at night
Decision Support for Doctors and Patients: A Scenario

Domain Knowledge
ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist;
*consider referral to specialist
Asthma Control and Actionable Information
Asthma Domain Knowledge

Did you cough more
than 20 times today?
- Yes
Semantic Sensor
Network Ontology
Did you cough more than 20
times today?
- Yes
times today?
- Yes
times today?
- Yes
times today?
- Yes
times today?
- No
Raw Data
Historical Data from a Person
20 times cough is http://knoesis.org/asthma/high-coughing
672 steps is http://knoesis.org/asthma/low-activity
1 h 17 mins REM Sleep is http://knoesis.org/asthma/disturbed-
sleep
Annotated Data 672 is http://knoesis.org/asthma#steps
1 h 17 mins is http://knoesis.org/asthma#REM_Sleep
20 is http://knoesis.org/asthma#Cough_Incident
Personalizati
on
Well Controlled
Very Poorly
Controlled
Not Well
Controlled
Contextualization
Abstractions
(Actionable Information)
Knowledge Base
and Unstructured Data
The symptoms (high-coughing, low-activity, disturbed-
sleep) are interpreted with respect to a person with
severity level “Mild Asthma”
Interpretation Exploration Understanding
CC
PC
SC
Example from kHealth Project

Another Example
with a Focus on a Richer Model

We are still working on the simpler representations of the real world!
http://artint.info/html/ArtInt_8.html, http://en.wikipedia.org/wiki/Traffic_congestion
Solve
Represent Interpret Real world
Simplified representation
Compute
What did not change in data processing for quite some time?

We need computational paradigms to tap into the rich pulse of the human
populace, and utilize diverse continuous stream of data
Represent, capture, and compute with richer and fine-grained representations of
real-world problems
Solve
Represent
Interpret Real world
Richer representation
Compute
+
Richer representation of
traffic observations.
Effective solutions
People interpreting a
real-world event.
What should change?

By 2001 over 285 million Indians lived in cities, more than in all North American
cities combined (Office of the Registrar General of India 2001) 1.
1 The Crisis of Public Transport in India
2 IBM Smarter Traffic
Modes of Transportation in Indian Cities
Texas Transportation Institute (TTI)
Congestion report in U.S.
Severity of the Traffic Problem

• What time to start?
• What route to take?
• What is the reason for traffic?
– Wait for some time or re-route?
Questions Asked Daily

Image credit: http://traffic.511.org/index
Multiple Events
Varying influence
Interact with Each Other
Challenge: Non-linearity in Traffic Dynamics

7 × 24
LDS(1,1), LDS(1,2) ,…., LDS(1,24)
LDS(7,1), LDS(7,2) ,…., LDS(7,24)
.
.
.
di
hj
Mon.
Tue.
Wed.
Thu.
Fri.
Sat.
Sun.
Mon.
Tue.
Wed.
Thu.
Fri.
Sat.
Sun.
Speed/travel-time time
series data from a link.
Time series data for each hour of
day (1-24) for each day of week
(Monday – Sunday).
Mean time series computed for
each day of week and hour of day
along with the medoid.
168 LDS models for each link;
Total models learned = 425,712
i.e., (2,534 links × 168 models
per link).
Step 1: Index data for each link for day of
week and hour of day utilizing the traffic
domain knowledge for piece-wise linear
approximation
Step 2: Find the “typical” dynamics by
computing the mean and choosing the
medoid for each hour of day and day of
week
Step 3: Learn LDS parameters for the
medoid for each hour of day (24 hours)
and each day of week (7 days) resulting
in 24 × 7 = 168 models for each link
Learning Context-specific LDS Models

Compute Log Likelihood for
each hour of observed data
(di,hj) LDS(hj,di)
7 × 24
Lik(1,1), Lik(1,2) ,…., Lik(1,24)
Lik(7,1), Lik(7,2) ,…., Lik(7,24)
.
.
.
Train?
Yes (Training phase)
Tag Anomalous hours using the Log
Likelihood Range
No
(di,hj) (min. likelihood)
Anomalies
L =
Partition based on (di,hj)
Speed and travel-time time
Observations from a link
Log likelihood min. and
max. values obtained from
five number summary
Partition based on (di,hj)
7 × 24
LDS(1,1), LDS(1,2) ,…., LDS(1,24)
LDS(7,1), LDS(7,2) ,…., LDS(7,24)
.
.
.
di
hj
(Input)
(Output)
Tagging Anomalies with LDS Models

Hourly Traffic Dynamics Over a Day

Public Safety
Urban Planning
Gov. & Agency
Admin.
Energy & water
Environmental
TransportationSocial Programs
Healthcare
Education
Twitter as a Source of City Events

Pramod Anantharam, Payam Barnaghi, Krishnaprasad Thirunarayan, and Amit Sheth. 2015. Extracting City Traffic Events from Social Streams.
ACM Trans. Intell. Syst. Technol. 6, 4, Article 43 (July 2015), 27 pages. DOI=10.1145/2717317 http://doi.acm.org/10.1145/2717317
Last O night O in O CA... O (@ O Half B-LOCATION Moon I-LOCATION Bay B-LOCATION
Brewing I-LOCATION Company O w/ O 8 O others) O http://t.co/w0eGEJjApY O
Extracting City Events from Textual Data

Complementary Events
Traffic Incident; road-construction
Textual Events from Tweets vs. 511.org: Complementary

Corroborative Events
Fog visibility-air-quality; fog
Textual Events from Tweets vs. 511.org: Corroborative

Timeliness
Concert Concert
Textual Events from Tweets vs. 511.org: Timeliness

Image Credit: http://traffic.511.org/index
slow-moving-traffic
Domain knowledge in the
form of traffic vocabulary
Domain knowledge of traffic flow
synthesized from sensor data
Explained-by
Horizontal operator: relating/mapping data from different
modality to a concept (theme) within a spatio-temporal
context;
Spatial context even include what it means to have a slow
traffic for the type of road
Understanding: Semantic Annotation of Sensor + Textual Data
Utilizing Background Knowledge

This example demonstrates use of:
– Multimodal data streams (types of events from text - signature from sensor data).
– Multiple sources of knowledge/ontologies.
– Semantic annotations and enrichments.
– Use of rich representation (PGM).
– Statistical approach to create normalcy models and understand anomalies using
historical data.
– Explain anomalies using extracted events.
– Provide actionable information.
How traffic analysis captures complexity of the real-world?

Domain Knowledge
Historical Data
Annotation
511.org
SC
CC
Slow traffic due
to football match.
Slow traffic due
to accident.
Slow traffic due
to construction.
raw numbers
Multimodal Data
Anomalous Traffic
Pattern
Learned Models
Abstractions
(Actional Information)
PC

Thank you, and please visit us at http://knoesis.org
For more information on kHealth, please visit us at
http://knoesis.org/projects/khealth
Cognitive
Computing
Semantic
Computing
Perceptual
Computing
Contributors and collaborators for this talk:
Pramod
Anantharam
Cory Henson Dr. T.K. Prasad
Sujan Perera Utkarshani Jaimini
Thank You

Perception can be split into two processes:
1.Processing sensory input, which transforms these low-level information to
higher-level information (e.g., extracts shapes for object recognition).
2. Processing which is connected with a person's concepts and expectations
(knowledge) and selective mechanisms (attention) that influence perception.
https://en.wikipedia.org/wiki/Perception
Perception

Scott Kelso uses “Circular Causality” to describe the situation in dynamic systems where
the cooperation of the individual parts of the system determine the global system’s
behavior, which in turn governs the behavior of the individual parts.
This is related to Andy Clark’s concept of continuous reciprocal causation, which occurs
when some system S is both continuously affecting and simultaneously being affected by
activity in some other system O.
https://mitpress.mit.edu/books/artificial-cognitive-systems Component Dynamics
Global System Behavior
InfluencesDetermine
Circular Causality and Autonomous Cognitive Systems

Perception is an active, cyclical
process of exploration and
interpretation.
“Perception and action, if these
unifying models are correct, are
intimately related and work
together to reduce prediction
error by sculpting and selecting
sensory inputs”
Perception-Action Cycle (Neisser, 1976)
https://www.amazon.com/Cognition-Reality-Principles-Implications-Psychology/dp/0716704773
Neisser Perceptual-Action Cycle

http://www.simplypsychology.org/perception-theories.html, http://www.youramazingbrain.org/supersenses/necker.htm
Psychologist Richard Gregory (1970) argued that perception is a constructive process which relies on top-down
processing.
A lot of information reaches the eye, but much is lost by the time it reaches the brain (Gregory estimates about
90% is lost). The brain has to guess what a person sees based on past experiences. Our perceptions of the world
are hypotheses based on past experiences and stored information (prior knowledge). Sensory receptors receive
information from the environment, which is then combined with previously stored information about the world
which we have built up as a result of experience. The formation of incorrect hypotheses will lead to errors of
perception
Richard Gregory’s Perception

http://www.people-clipart.com/people_clipart_images/clip_art_illustration_of_a_young_woman_with_her_hand_on_her_hip_wearing_jeans_and_a_half_sleeve_shirt_0071-0908-
3022-3627.html Zadeh, Lotfi A. "Toward a perception-based theory of probabilistic reasoning with imprecise probabilities." Journal of statistical planning and inference 105, no. 1
(2002): 233-264 http://sweetclipart.com/cute-little-girl-holding-daisy-719 http://www.keyword-suggestions.com/aGlzcGFuaWMgYXJ0/
Lotfi Zadeh and Perception
• Measurements are crisp;
perceptions are fuzzy.
• Perception-based information is
drawn from natural language.
• Humans use perceptions of time,
direction, speed, shape, possibility,
likelihood, truth, and other
attributes of physical and mental
objects.
• Perceptions are imprecise.
• Example: not very high, about 0.8

1. Cognition is a general term for all forms of knowing (e.g. attending,
remembering, reasoning and understanding concepts, facts, propositions, and
rules).
2. Cognitive processes are how you manipulate your mental contents.
3. Cognitive psychology is the study of cognition.
4. Cognitive science is an interdisciplinary field that extends the principles of
cognitive psychology to other systems that manipulate information.
Cognition and Brain-Non-invasive brain scanning allows correlations to be made between
human conscious experiences and patterns of brain activity. Studies of both visual and
auditoryperception allow distinctions to be made between brain regions that do and do not
show activity patterns that correlate with conscious experiences. Results from study of brain
lesions, application of drugs, and electromagnetic disruption of the function of specific brain
regions can be interpreted in combination with results from brain scans.
1. Neural correlates of the visual vertical meridian asymmetry by Taosheng Liu, David J. Heeger, and Marisa Carrasco in Journal of Vision (2007)
Volume 6: 1294–1306.
2. Jump up Hierarchical Processing of Auditory Objects in Humans by Sukhbinder Kumar, Klaas E Stephan, Jason D Warren, Karl J Friston and Timothy D Griffithsin in PLoS Comput
Biol. (2007) Volume 3:e100.
https://en.wikiversity.org/wiki/Cognition
What does cognition mean? (from Wikipedia)

Cognition enables individuals to understand their environment and paves
the way for the application of perception to explore and deepen the
understanding
The network of concepts and relationships enables the cognition and
perception needed to interpret daily experiences.
What does cognition mean? (from Wikipedia)

The system’s understanding of it’s worlds is inherently specified to the
form of the systems’ embodiment and is dependent on the system’s
history of interactions and it’s experiences.
So this property of making sense of it’s environmental interactions is one
of the foundations of a branch of cognitive science called “Enaction”
From https://mitpress.mit.edu/books/artificial-cognitive-systems
Emergent Paradigm of Cognitive Science

From http://philosophyofbrains.com/2015/12/14/surfing-uncertainty-prediction-action-and-the-embodied-mind.aspx
Top-Down and Bottom-Up (Andy Clark)

http://www.fil.ion.ucl.ac.uk/~karl/Whatever%20next.pdf, http://libertymotive.com/truth/
Top-Down Approach model emphasizes the use of background knowledge to predict content.
The information that needs to be communicated ‘upward’ is just the prediction error: the
divergence from the expected signal.
Top-down Expectation/Prediction
firm, not squishy
red and green, not just red
Prediction Error
fruit
stem
Bottom-up Observation
Andy Clark

Attempts to build an artificial cognitive system that can be positioned in a two-dimensional space, with one axis
defining a spectrum running from purely computational techniques to techniques strongly inspired by biological
models, and with another axis defining the level of abstraction of the biological model.
Adapted from https://mitpress.mit.edu/books/artificial-cognitive-systems.
Inspiration
Abstraction Level
Modelling decomposition of
hypothetical model of brain
Cognitive system modelled on
the microscopic organization of
the brain
Cognitive system based on
statistical learning of
specific domain rules
Cognitive system based on
artificial neural networks
Computational Biological
Low
High
IBM Brain Chip
Aspects of Modelling Cognitive Systems

Understanding traffic flow variations

Histogram of speed values
collected from June 1st 12:00 AM to June 2nd 12:00 AM
Histogram of travel time values
collected from June 1st 12:00 AM to June 2nd 12:00 AM
Traffic Data: First Peek

This distribution resembles a Gaussian Mixture
Model (GMM)
Multiple Gaussian Distributions: A Better Fit for Speed Observations?

Assume Normalcy to be uninterrupted traffic flow
July 2014 has no events so, we
hypothesize higher log-likelihood
score
June 2014 has many events so, we
hypothesize lower log-likelihood
score
-115655.8
(Closer to Normalcy)
-125974.3
88
Golden Gate Fields: Comparing Months with Varying Event Occurrences

Root Cause Analysis Action Recommendation
Find Triggers of Asthma
Derive the cause of asthma attacks
for a given patient using statistical
techniques + knowledge of asthma
and its triggers.
Minimize Asthma Attacks
Model actions based on the utility
theory (cost of actions & its rewards)
+ knowledge of action consequences.
Two Research Directions for kHealth Asthma with More Ddata…

Sensordrone
• Precision gas sensor
• Reducing gas sensor
• Oxidizing gas sensor
• Non-contact
thermometer
• Humidity sensor
• Temperature sensor
• Light sensor
• Color sensors
• Pressure
• Proximity
• Expansion connector
Node Sensor
• Exhaled Nitric Oxide
Fitbit ChargeHR
• Heart Rate
• All-Day Activity
• Sleep Monitoring
90
kHealth Kit: Sensor Platforms

• If an anomaly is detected on a link L and during time period [tst, tet], then the
anomaly is explained by an event if the event occurred in the vicinity within 0.5km
radius and during [tst-1, tet+1].
• CAVEAT: An anomaly may not be explained because of missing data.
Thanks to Dr. Krishnaprasad Thirunarayan for sharing this slide.
Spatio-temporal Co-occurrence Criteria

Brain to Computing
Second Approach to Intelligence

How can we get inspiration
from brain to computing?

One approach, recreate the brain in
silicon. A second approach, different
disciplines learn about the human brain,
get inspiration and learn things for
computing. (Slide on mapping from brain
to computing.)

"The domain of cognitive science occupies the intersection of philosophy, neuroscience,
linguistics, cognitive psychology, and computer science (artificial intelligence)."
Gerrig, R. J., Zimbardo, P. G., Campbell, A. J., Cumming, S. R., & Wilkes, F. J. (2008). Psychology and life (Australian edition). Sydney: Pearson Education
Australia, p. 248.https://en.wikiversity.org/wiki/Cognitive_science.
Cognitive Science

https://mitpress.mit.edu/books/artificial-cognitive-systems
Two hallmarks of cognition:
● Prediction out of the past interactions.
● Learning new knowledge by making sense of its interactions with the
world around it.
The dependency on exploration and development is one of the reasons why
an artificial system requires a rich sensory-motor interface with its
environment and why embodiment plays such a pivotal role.
Hallmarks of Cognition

Contextual
https://en.wikipedia.org/wiki/Cognitive_computing http://edtechreview.in/dictionary/174-what-is-adaptive-learning
http://www.ibm.com/developerworks/rational/library/may05/bittner-spence/ http://www.inspiring-visuals.com/interactive-multimedia-presentations.shtml
http://leadingagent.net/blog/an-awesome-contextual-real-estate-crm-solution-is-critical-to-smart-real-estate-practice-growth/
Features that cognitive systems express are:
Adaptive
Interactive
Iterative and stateful
Cognitive Systems

Reduced
CO level =>
better asthma
control
High CO influences
Wheezing Level (Low/High)
High CO High Luminosity
High Wheeze Low Luminosity
Low Wheeze
Carbon Monoxide
Wheeze
Luminosity
1Amit Sheth, Pramod Anantharam, Cory Henson, 'Physical-Cyber-Social Computing: An Early 21st Century Approach,' IEEE Intelligent Systems, vol. 28, no. 1, pp. 78-82,
Jan.-Feb., 2013. http://doi.ieeecomputersociety.org/10.1109/MIS.2013.20
Horizontal Operators
(Semantic Integration) operates
on data from heterogeneous
sources to create
Integrated or correlated
data streams.
Vertical Operators
(Semantic abstraction) operates on
artifacts at each level and transcends
them to the next level.
“a holistic treatment of data, information,
and knowledge from the PCS worlds to
integrate, correlate, interpret, and
provide contextually relevant abstractions to
humans.”1
Physical-Cyber-Social Computing for Actionable Insights from Multimodal Data

http://phys.org/news/2014-10-nobel-importance.html https://student.societyforscience.org/article/nobel-goes-finding-brain%E2%80%99s-%E2%80%98gps%E2%80%99
http://www.scilogs.be/neuropolis/hoe-helpen-de-kanaaltjes-in-ons-oor-ons-om-te-navigeren/

https://mitpress.mit.edu/books/soar-cognitive-architecture, http://www.hindawi.com/journals/mpe/2012/530561/fig1/
Soar Cognitive Architecture

http://my.ilstu.edu/~jrbaldw/370/Meaning.htm
Semiotics

Brain GPS - Grid Cell/ Place Cells
● in 1971 John O’Keefe of University College London discovered “place cells” found in
Hippocampus.
● In 2005 May-Britt and Edvard Moser of Norwegian University of Science and Technology added a
new discovery : the existence of “grid cells” in the nearby cortex.
● The grid and place cells work together to constitute an internal navigation system.
● Grid cells provide a set of coordinates that enable a rat to navigate through its environment in
conjunction with other cells that recognize the positioning of the head and the borders of a room.
● Grid cell systems give you immediate sense of recognition of, yes, this is where I am. "That's my
home, there's my office, there's the entrance to the metro."
● Alzheimer's patients wander off into the night, as a consequence of the dying of neurons in the
entorhinal cortex and the hippocampus.
https://www.kth.se/blogs/prasanth/tag/india-and-sweden/ http://www.dallasnews.com/opinion/sunday-commentary/20130920-gps-may-reroute-the-brain.ece
http://www.scientificamerican.com/sciam/assets/media/multimedia/110514-MindListicle/1.jpg http://www.activebeat.com/health-news/nobel-prize-awarded-to-scientists-
who-discovered-brain-gps-system/ http://www.scientificamerican.com/article/10-big-ideas-in-10-years-of-brain-science/

http://www.scientificamerican.com/article/10-big-ideas-in-10-years-of-brain-science/
In the future optogenetics will allow us to decipher both how various
brain cells elicit feelings, thoughts and movements—as well as how
they can go awry to produce psychiatric disorders.
● Stanford scientists presented a technique for switching individual
neurons on or off with light in 2005.
● To probe how a certain class of neurons helps mice navigate
mazes, scientists insert electrodes into brain tissue and stimulate
thousands of neurons at a time.
● Scientists can tuck light-sensitive molecules into specific brain
cells to manipulate only those selected neuron types or networks.
● Shining a light makes those neurons either more or less active
and can elucidate their role in a behavior or disease.Mouse with optogenetic tools in operation, including implanted fiberoptic
and light-sensitive molecules produced in the brain, all representing
technologies developed in the Deisseroth lab at Stanford University by
graduate students Raag Airan, Feng Zhang, Ed Boyden, and Lief Fenno.
Credit: Raag Airan, Feng Zhang, Ed Boyden, and Lief Fenno
“Over the past decade hundreds of research groups have used optogenetics to learn how various networks of neurons
contribute to behavior, perception and cognition,” wrote Ed Boyden, a co-inventor of optogenetics.
Optogenetics

http://www.scientificamerican.com/sciam/assets/media/multimedia/110514-MindListicle/1.jpg http://www.scientificamerican.com/article/10-big-ideas-in-10-years-of-brain-science/
http://www.nature.com/nrn/journal/v5/n7/fig_tab/nrn1429_F1.html
Woman at the Human Genome exhibit at the National Museum of Natural History in Washington, D.C.
Human Genome Project (HGP)
Mapping and Identification of clusters of disease-related human genes for boosting our
understanding of genetic pathways that spawn neurological and psychiatric disorders
like schizophrenia, Alzheimer’s disease, depression and autism, among other disorders.
“Neurogenetics is the role of genetics in the development and function of the nervous system”
Neurogenetics

Philanthropist Paul Allen, co-founder of Microsoft
● In 2003 , Allen Institute of Brain Science began mapping gene activity in
mouse brain
● Genetic activity help researchers discover genes relevant to certain
diseases or behaviors.
● Institute launched a 10-year plan to examine where specific genes are
active and how these genetic circuits process the vast flow of
information into the brain.
● For example, the brain map showed that genes associated with autism
appear to be acting on a specific type of brain cell in neocortex. That
suggests "we should be looking at this particular type of cell in the
neocortex, and furthermore that we should probably be looking very
early in the prenatal stages for the origin of autism".
A top-down 3-D view of the cortico-connections
originating from multiple distinct cortical areas,
visualized as virtual tractography using Allen
Institute Brain Explorer software. Credit: Allen
Institute for Brain Science.
http://www.scientificamerican.com/sciam/assets/media/multimedia/110514-MindListicle/1.jpg
Brain Mapping

Neurocognitive functions are cognitive functions linked with the functions of
particular areas, neural pathways, or cortical networks in the brain substrate layers
of neurological matrix at the cellular molecular level.
Their understanding is closely linked to the neuropsychology and cognitive
neuroscience, two disciplines that broadly seek to understand how the structure
and function of the brain relates to perception, de-fragmentation of concepts,
memory embed, association and recall both in the thought process and behavior.
https://en.wikipedia.org/wiki/Neurocognitive
Neurocognitive

The term 'cognitive neuroscience' was coined by George Miller and Michael Gazzaniga toward the end of the 1970s.
Cognitive neuroscience began to integrate the newly laid theoretical ground in cognitive science, that emerged
between the 1950s and 1960s.
Cognitive neuroscience is concerned with the scientific study of the biological processes and aspects that underlie
cognition, with a specific focus on the neural connections in the brain which are involved in mental processes.
It addresses the questions of how psychological/cognitive activities are affected or controlled by neural circuits in the
brain. Cognitive neuroscience relies upon theories in cognitive science coupled with evidence from neuropsychology,
and computational modeling.
Neurons play the most vital role, since the main point is to establish an understanding of cognition from a neural
perspective, along with the different lobes of the cerebral cortex.
David Marr (neuroscientist) concluded that one should understand any cognitive process at three levels of analysis:
• Computational.
• Algorithmic/representational.
• Physical levels of analysis.
https://en.wikipedia.org/wiki/Cognitive_neuroscience
Cognitive Neurocognitive

http://blogs.scientificamerican.com/talking-back/discovery-of-brain-s-navigation-system-wins-2014-nobel-prize-in-medicine/
References

Semantic, Cognitive and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Semantic, Cognitive and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing