See http://grammars.grlmc.com/WebST2016/courseDescription.php#Amit_keynote
While Bill Gates, Stephen Hawking, Elon Musk, Peter Thiel and others engaged in OpenAI discuss whether or not AI, robots, and machines will replace humans, proponents of human-centric computing continue to extend work in which humans and machine partner in contextualized and personalized processing of multimodal data to derive actionable information. In this talk, we discuss how maturing paradigms such as semantic computing (SC), cognitive computing (CC), complemented by the emerging perceptual computing (PC) paradigm provide a continuum through which to exploit the ever-increasing and growing diversity of data that could enhance people’s daily lives. SC and CC sift through raw data to personalize it according to context and individual user, creating abstractions that move the data closer to what humans can readily understand and apply in decision-making. PC, which interacts with the surrounding environment to collect data that is relevant and useful in understanding the outside world, is characterized by interpretative and exploratory activities, that is supported by use of prior/background knowledge. Using the examples of personalized digital health and smart city, we will demonstrate how SC, CC and PC form complementary capabilities that will enable development of next generation of intelligent systems.
References:
Amit Sheth, "Computing for Human Experience: Semantics-Empowered Sensors, Services, Social Computing on the Ubiquitous Web," IEEE Internet Computing, 14 (1), January/February 2010.
Amit Sheth, Pramod Anantharam, Cory Henson, Semantic, Cognitive, and Perceptual Computing: Advances toward Computing for Human Experience,IEEE Computer, March 2016. http://online.qmags.com/CMG0316/default.aspx?pg=67&mode=2#pg67&mode2
Amit Sheth, Internet of Things to Smart IoT Through Semantic, Cognitive, and Perceptual Computing, IEEE Intelligent Systems, March/April 2016.
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Semantic, Cognitive and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing
1. Amit Sheth
Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing:
Wright State University, Dayton, Ohio
Semantic, Cognitive, and Perceptual Computing:
three intertwined strands of a golden braid of intelligent computing
Keynote @ International Summer School on Web Science & Technology (WebST), Bilbao, Spain, 19 May 2016.
3. Credit: Looi Consulting (http://www.looiconsulting.com/home/enterprise-big-data/)
● In 2008, data generated > storage available. Less than 0.5% of data get analyzed.
● Vast variety of data: text > images > A/V > genome sequencing > IoT
● Of all the data generated, which data is relevant, and why? Which data to analyze? Which
data can offer insight? Who cares for what data? How to get attention to a human decision
maker? What we need is intelligent processing to get actionable, smart data.
A Big Challenge and Opportunity in Recent Times
4. How would an enterprise get actionable information?
http://www.slideshare.net/NamrataChatterjee/nokias-supply-chain-management-case-study, http://www.economist.com/node/7032258
● Weak crisis judgement.
● Failure to take prompt action.
● Single supplier reliability.
Fire at Royal Philips electronic
semiconductor plant, New
Mexico in March 2000.
8 trays of wafers
containing the
miniature circuitry to
make several
thousand chips for
mobile phones was
destroyed.
The expected time to recover was estimated to be a week.
● Fire breakout in clean room.
● Inability to determine the exact
damage to the clean room.
● Lack of emergency preparation.
● Early speculation of possible crisis.
● Preparedness against supply crisis.
● Finding alternative source of chip
supply.
6. First used in 2004 redefined 2013: http://wiki.knoesis.org/index.php/Smart_Data.
Smart Data
7. ● The astounding bandwidth of your senses is 11
million bits of information every second.
● In conscious activities like reading, the human
brain distills approximately 40 bits of
information per second.
http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.html
The Brain: Inspiration for Intelligent Processing:
What if we could automate such interpretation of data?
8. ● How can we take inspiration from human brain and derive an intelligent processing of big data
○ for service enterprise need to serving individual needs
● In particular, take inspiration from cognition and perception, and for pedagogy, outline three
computing paradigms for building future intelligent systems focused on converting data into
abstractions that humans act upon for making decision and taking action
○ not all aspects of capturing human intelligence are addressed
○ not focused on making machines that replace humans - instead focus is on CHE
● CHE: Humans and machines partner to enhance human experience and better informed decision
making with humans in the Loop.
* 2010 article: http://wiki.knoesis.org/index.php/Computing_For_Human_Experience; distinct from “Human Experience of Computing by Booch:
Dec. 9, 2011 at http://computingthehumanexperience.com/category/computing/.
Sheth - 2008+ *
What is this talk about?
9. http://www.livescience.com/1863-theory-intelligence-works.html, http://www.wired.com/2015/10/scientists-can-now-predict-intelligence-brain-activity/
Human Intelligence is not confined to
a single area in the brain, but is the
result of multiple brain areas working
in coordination.
Human Intelligence is related to how
information travels through the brain.
According to a review of 37 imaging studies in Journal Behavioral and Brain
Science, intelligence is related not so much to brain size or brain structure,
but to how efficiently information travels through the brain.
Some brains are better than others at
certain things because of the way they are
wired.
“The more certain regions are talking to
one another, the better you are able to
process information quickly and make
inference.”
Intelligence Research
13. Prior Medical Knowledge
D1
D3
D1
Medical History and
Past observations
S1
S2
S3
Sn
..
..
D1
D2
D3
S1
S2
S3
..
..
..
..
..
..
..
..
D1
D1
Doctor Patient
Q1
Q2
Qn
A1
A2
An
Blood Pressure
Heart Rate
Breathing Rate
Body Temperature
Multi-model Observations
Current: Observing a Snapshot of the Patient
14. ACTIONS
situation awareness useful
for decision making
ABSTRACTIONS
make sense to humans
KNOWLEDGE
for interpretation of observations
Contextualization
Personalization
DATA
Observations from machine
and social sensors
Converting Data to Actions
15. 1Marcus, Philip, Kevin R. Murphy, Abid Rahman, and Christopher D. O’Brien. "Intrapatient symptom variability in
adults and children with asthma: Results of a survey." Advances in therapy 22, no. 5 (2005): 488-497.
“ … survey indicates that adult patients and caregivers of pediatric patients
report variability in asthma symptoms over time, even when asthma medications are taken.”1
Personal level
Signals
Public level
Signals
Population level
Signals
Future: Analyzing a Multifaceted Continuous Stream of Diverse Data
16. How do we solve problems with real-world complexity, gather vast amounts of data,
diverse knowledge, and come up with intelligent decisions and timely actions?
Next, a pedagogical approach.
17.
18. Semantics, perception, and cognition interact seamlessly.
● Semantic Computing can deal with big data challenges.
● Cognitive Computing can use relevant knowledge to improve data
understanding for decision-making.
● Perceptual Computing can provide personalized and contextual abstractions
over massive amounts of multimodal data from the physical, cyber, and social
realms.
https://www.linkedin.com/pulse/perceptual-computing-third-strand-golden-braid-amit-sheth
Semantic Computing, Perceptual Computing, Cognitive Computing
20. Semantics attaches meaning to observation by providing a definition within a system context
or the knowledge that people possess [Sheth 2016].
Semantic Computing encompasses the technology required to represent concepts and their
relationships in an integrated semantic network that loosely mimics the brain’s conceptual
interrelationships.
Web of data
Semantics attached to
objects in the world
Semantic Computing
21. Population Level
Personal
Wheeze – Yes
Do you have tightness of chest? –Yes
ObservationsPhysical-Cyber-Social System Health Signal Extraction Health Signal Understanding
<Wheezing=Yes, time, location>
<ChectTightness=Yes, time, location>
<PollenLevel=Medium, time, location>
<Pollution=Yes, time, location>
<Activity=High, time, location>
Wheezing
ChectTightness
PollenLevel
Pollution
Activity
Wheezing
ChectTightness
PollenLevel
Pollution
Activity
RiskCategory
<PollenLevel, ChectTightness, Pollution,
Activity, Wheezing, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
.
.
.
Expert
Knowledge
Background
Knowledge
tweet reporting pollution level
and asthma attacks
Acceleration readings from
on-phone sensors
Sensor and personal
observations Signals from personal,
personal spaces, and
community spaces
Risk Category assigned by
doctors
Qualify
Quantify
Enrich
Outdoor pollen and pollution
Public Health
Well Controlled - continue
Not Well Controlled – contact nurse
Poor Controlled – contact doctor SSN
Semantic Computing for Our Personalized Digital Health (Asthma) Application
22. Cognition is the process by which an autonomous system perceives its
environment, learns from experiences, anticipates the outcome of the events,
acts to pursue goals and adapt to the changing environment.
https://mitpress.mit.edu/books/artificial-cognitive-systems
PerceptionAction
Anticipate
Adapt Assimilate
Cognition as a cycle of anticipation, assimilation,
adaptation: embedded in, contributing to, and benefitting
from a continuous process of action and perception.
What is Cognition?
23. DARPA launched a Cognitive Computing project in 2002. Cognitive Computing was defined as
the ability to “reason, use represented knowledge, learn from experience, accumulate
knowledge, explain itself, accept direction, be aware of its own behavior and capabilities,
[and] respond in a robust manner to surprises.”
IBM describes the components used to develop, and behaviors resulting from, “systems that
learn at scale, reason with purpose and interact with humans naturally.” According to them,
while sharing many attributes with the field of artificial intelligence, it differentiates itself via
the complex interplay of disparate components, each of which comprise their own individual
mature disciplines [1].
Our Take
Cognitive computing interprets annotated observations obtained from Semantic computing ,
or raw observations from diverse sources and presents actionable information to humans.
Cognitive computing systems learns from their experiences and improve when performing
repeated tasks.
[1] https://en.wikipedia.org/wiki/Cognitive_computing
Cognitive Computing
25. “where we see the world we do not decide to see it” Daniel Kahneman
Perception is an active cyclical process of exploration and
interpretation.
Perception enables individual to focus on most promising
course of action by incorporating background knowledge that
provided a comprehensive contextual understanding
Perception
27. Perceptual computing is the ability for a computer to recognize what is going on around it.
Computer can perceive the environment and the users in that environment. The computer
determines what needs a user might have and react to those needs without giving or
receiving any additional information.
Perception: Intel
29. Perceptual Computing supports the ability to:
● Ask contextually relevant and personalized questions.
● Complement Semantic Computing and Cognitive Computing by providing the machinery to
ask the next question or derive a hypothesis.
It uses machine perception together with available background knowledge to explore and
interpret observations.
Perceptual Computing system enables the personalization of information provided by the
cognitive computing system.
For example, it minimizes the uncertainty by providing a personalized and contextualized
understanding of a patient’s environment and symptomatic variations.
Perceptual Computing: Our View
31. Making Sense of Sensor Data With
Henson, et al. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web, Applied Ontology, 2011.
Our First Approach to Implement Perceptual Computing
34. Explanation is the act of choosing the objects or events that best account for a set of
observations; often referred to as hypothesis building
Discrimination is the act of finding those properties that, if observed, would help distinguish
between multiple explanatory features
1
2
Explanation and Discrimination
35. Explanatory Feature: a feature that explains the set of observed properties
elevated blood pressure Hypertension
Hyperthyroidism
Pulmonary Edema
Clammy Skin
Palpitations
Explanation
36. Discriminating Property: is neither expected nor not-applicable
clammy skin
Hypertension
Hyperthyroidism
Pulmonary EdemaPalpitations
elevated blood pressure
Discrimination
37. Orders of magnitude resource savings for generating and storing relevant abstractions vs. raw observations.
Relevant Abstractions
Raw Observations
Resource Savings of Abstracting Sensor Data
39. Providing actionable information in a timely manner is
crucial to avoid information overload or fatigue
Sleep data Community dataPersonal
Schedule
Activity data Personal health
records
Data Overload for Patients/Health Aficionados
40. Personal level
Signals
Public level
Signals
Population level
Signals
Domain
Knowledge
http://www.tuberktoraks.org/managete/fu_folder/2011-03/html/2011-3-291-311.html
Contextual ActionablePersonalized
OR
How is my Asthma control?
Should I take additional medication today?
How can I reduce my asthma attacks at home?
Asthma: Challenges in Heterogeneity, Variability, and Personalization
41. Sensordrone
(Carbon monoxide,
temperature, humidity)
Sensor Platforms
Android Device
(w/ kHealth
App)
Total cost: ~ $550
Along with sensor platforms in the kit, the application uses a variety of population
level signals from the web:
Pollen level Air Quality Temperature & Humidity
Node Sensor
(exhaled Nitric Oxide)
Fitbit ChargeHR
(Activity, sleep quality)
kHealth Kit for the Application for Asthma Management
42. For collecting observations from both machine sensors
and from patients in the form of a questionnaire.
kHealth Kit: Android Application
44. Personal level
Signals
Public level
Signals
Population level
Signals
Domain
Knowledge
Risk Model
Events from
social streams
Take medication before
going to work.
Avoid going out in the
evening due to high pollen
levels.
Contact a doctor.
Analysis
Personalized
Actionable
Information
Data Acquisition &
aggregation
kHealth: Health Signal Processing Architecture
46. Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
Patient health
Score
How vulnerable* is my control level today?
*considering changing environmental conditions and current control level
Patient Health Score (Prognostic)
47. Sensordrone – for monitoring
environmental air quality
Wheezometer – for monitoring
wheezing sounds
Can I reduce my asthma attacks at night?
What are the triggers? What is the wheezing level?
What is the propensity toward asthma?
What is the exposure level over a day?
Commute to work
Luminosity
CO Level
CO in gush during
the daytime.
Actionable
Information
Personal level
Signals
Public level
Signals
Population level
Signals
What is the air quality indoors?
Close the window at home during day to avoid
CO2 inflow, to avoid asthma attacks at night
Decision Support for Doctors and Patients: A Scenario
48. Domain Knowledge
ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist;
*consider referral to specialist
Asthma Control and Actionable Information
Asthma Domain Knowledge
49. Did you cough more
than 20 times today?
- Yes
Semantic Sensor
Network Ontology
Did you cough more than 20
times today?
- Yes
Did you cough more than 20
times today?
- Yes
Did you cough more than 20
times today?
- Yes
Did you cough more than 20
times today?
- Yes
Did you cough more than 20
times today?
- No
Raw Data
Historical Data from a Person
20 times cough is http://knoesis.org/asthma/high-coughing
672 steps is http://knoesis.org/asthma/low-activity
1 h 17 mins REM Sleep is http://knoesis.org/asthma/disturbed-
sleep
Annotated Data 672 is http://knoesis.org/asthma#steps
1 h 17 mins is http://knoesis.org/asthma#REM_Sleep
20 is http://knoesis.org/asthma#Cough_Incident
Personalizati
on
Well Controlled
Very Poorly
Controlled
Not Well
Controlled
Contextualization
Abstractions
(Actionable Information)
Knowledge Base
and Unstructured Data
The symptoms (high-coughing, low-activity, disturbed-
sleep) are interpreted with respect to a person with
severity level “Mild Asthma”
Interpretation Exploration Understanding
CC
PC
SC
Example from kHealth Project
51. We are still working on the simpler representations of the real world!
http://artint.info/html/ArtInt_8.html, http://en.wikipedia.org/wiki/Traffic_congestion
Solve
Represent Interpret Real world
Simplified representation
Compute
What did not change in data processing for quite some time?
52. We need computational paradigms to tap into the rich pulse of the human
populace, and utilize diverse continuous stream of data
Represent, capture, and compute with richer and fine-grained representations of
real-world problems
Solve
Represent
Interpret Real world
Richer representation
Compute
+
Richer representation of
traffic observations.
Effective solutions
People interpreting a
real-world event.
What should change?
53. By 2001 over 285 million Indians lived in cities, more than in all North American
cities combined (Office of the Registrar General of India 2001) 1.
1 The Crisis of Public Transport in India
2 IBM Smarter Traffic
Modes of Transportation in Indian Cities
Texas Transportation Institute (TTI)
Congestion report in U.S.
Severity of the Traffic Problem
54. • What time to start?
• What route to take?
• What is the reason for traffic?
– Wait for some time or re-route?
Questions Asked Daily
57. 7 × 24
LDS(1,1), LDS(1,2) ,…., LDS(1,24)
LDS(7,1), LDS(7,2) ,…., LDS(7,24)
.
.
.
di
hj
Mon.
Tue.
Wed.
Thu.
Fri.
Sat.
Sun.
Mon.
Tue.
Wed.
Thu.
Fri.
Sat.
Sun.
Speed/travel-time time
series data from a link.
Time series data for each hour of
day (1-24) for each day of week
(Monday – Sunday).
Mean time series computed for
each day of week and hour of day
along with the medoid.
168 LDS models for each link;
Total models learned = 425,712
i.e., (2,534 links × 168 models
per link).
Step 1: Index data for each link for day of
week and hour of day utilizing the traffic
domain knowledge for piece-wise linear
approximation
Step 2: Find the “typical” dynamics by
computing the mean and choosing the
medoid for each hour of day and day of
week
Step 3: Learn LDS parameters for the
medoid for each hour of day (24 hours)
and each day of week (7 days) resulting
in 24 × 7 = 168 models for each link
Learning Context-specific LDS Models
58. Compute Log Likelihood for
each hour of observed data
(di,hj) LDS(hj,di)
7 × 24
Lik(1,1), Lik(1,2) ,…., Lik(1,24)
Lik(7,1), Lik(7,2) ,…., Lik(7,24)
.
.
.
Train?
Yes (Training phase)
Tag Anomalous hours using the Log
Likelihood Range
No
(di,hj) (min. likelihood)
Anomalies
L =
Partition based on (di,hj)
Speed and travel-time time
Observations from a link
Log likelihood min. and
max. values obtained from
five number summary
Partition based on (di,hj)
7 × 24
LDS(1,1), LDS(1,2) ,…., LDS(1,24)
LDS(7,1), LDS(7,2) ,…., LDS(7,24)
.
.
.
di
hj
(Input)
(Output)
Tagging Anomalies with LDS Models
60. Public Safety
Urban Planning
Gov. & Agency
Admin.
Energy & water
Environmental
TransportationSocial Programs
Healthcare
Education
Twitter as a Source of City Events
61. Pramod Anantharam, Payam Barnaghi, Krishnaprasad Thirunarayan, and Amit Sheth. 2015. Extracting City Traffic Events from Social Streams.
ACM Trans. Intell. Syst. Technol. 6, 4, Article 43 (July 2015), 27 pages. DOI=10.1145/2717317 http://doi.acm.org/10.1145/2717317
Last O night O in O CA... O (@ O Half B-LOCATION Moon I-LOCATION Bay B-LOCATION
Brewing I-LOCATION Company O w/ O 8 O others) O http://t.co/w0eGEJjApY O
Extracting City Events from Textual Data
65. Image Credit: http://traffic.511.org/index
slow-moving-traffic
Domain knowledge in the
form of traffic vocabulary
Domain knowledge of traffic flow
synthesized from sensor data
Explained-by
Horizontal operator: relating/mapping data from different
modality to a concept (theme) within a spatio-temporal
context;
Spatial context even include what it means to have a slow
traffic for the type of road
Understanding: Semantic Annotation of Sensor + Textual Data
Utilizing Background Knowledge
66. This example demonstrates use of:
– Multimodal data streams (types of events from text - signature from sensor data).
– Multiple sources of knowledge/ontologies.
– Semantic annotations and enrichments.
– Use of rich representation (PGM).
– Statistical approach to create normalcy models and understand anomalies using
historical data.
– Explain anomalies using extracted events.
– Provide actionable information.
How traffic analysis captures complexity of the real-world?
67. Domain Knowledge
Historical Data
Annotation
511.org
SC
CC
Slow traffic due
to football match.
Slow traffic due
to accident.
Slow traffic due
to construction.
raw numbers
Multimodal Data
Anomalous Traffic
Pattern
Learned Models
Abstractions
(Actional Information)
PC
68.
69. Thank you, and please visit us at http://knoesis.org
For more information on kHealth, please visit us at
http://knoesis.org/projects/khealth
Cognitive
Computing
Semantic
Computing
Perceptual
Computing
Contributors and collaborators for this talk:
Pramod
Anantharam
Cory Henson Dr. T.K. Prasad
Sujan Perera Utkarshani Jaimini
Thank You
73. Perception can be split into two processes:
1.Processing sensory input, which transforms these low-level information to
higher-level information (e.g., extracts shapes for object recognition).
2. Processing which is connected with a person's concepts and expectations
(knowledge) and selective mechanisms (attention) that influence perception.
https://en.wikipedia.org/wiki/Perception
Perception
74. Scott Kelso uses “Circular Causality” to describe the situation in dynamic systems where
the cooperation of the individual parts of the system determine the global system’s
behavior, which in turn governs the behavior of the individual parts.
This is related to Andy Clark’s concept of continuous reciprocal causation, which occurs
when some system S is both continuously affecting and simultaneously being affected by
activity in some other system O.
https://mitpress.mit.edu/books/artificial-cognitive-systems Component Dynamics
Global System Behavior
InfluencesDetermine
Circular Causality and Autonomous Cognitive Systems
75. Perception is an active, cyclical
process of exploration and
interpretation.
“Perception and action, if these
unifying models are correct, are
intimately related and work
together to reduce prediction
error by sculpting and selecting
sensory inputs”
Perception-Action Cycle (Neisser, 1976)
https://www.amazon.com/Cognition-Reality-Principles-Implications-Psychology/dp/0716704773
Neisser Perceptual-Action Cycle
76. http://www.simplypsychology.org/perception-theories.html, http://www.youramazingbrain.org/supersenses/necker.htm
Psychologist Richard Gregory (1970) argued that perception is a constructive process which relies on top-down
processing.
A lot of information reaches the eye, but much is lost by the time it reaches the brain (Gregory estimates about
90% is lost). The brain has to guess what a person sees based on past experiences. Our perceptions of the world
are hypotheses based on past experiences and stored information (prior knowledge). Sensory receptors receive
information from the environment, which is then combined with previously stored information about the world
which we have built up as a result of experience. The formation of incorrect hypotheses will lead to errors of
perception
Richard Gregory’s Perception
77. http://www.people-clipart.com/people_clipart_images/clip_art_illustration_of_a_young_woman_with_her_hand_on_her_hip_wearing_jeans_and_a_half_sleeve_shirt_0071-0908-
3022-3627.html Zadeh, Lotfi A. "Toward a perception-based theory of probabilistic reasoning with imprecise probabilities." Journal of statistical planning and inference 105, no. 1
(2002): 233-264 http://sweetclipart.com/cute-little-girl-holding-daisy-719 http://www.keyword-suggestions.com/aGlzcGFuaWMgYXJ0/
Lotfi Zadeh and Perception
• Measurements are crisp;
perceptions are fuzzy.
• Perception-based information is
drawn from natural language.
• Humans use perceptions of time,
direction, speed, shape, possibility,
likelihood, truth, and other
attributes of physical and mental
objects.
• Perceptions are imprecise.
• Example: not very high, about 0.8
78. 1. Cognition is a general term for all forms of knowing (e.g. attending,
remembering, reasoning and understanding concepts, facts, propositions, and
rules).
2. Cognitive processes are how you manipulate your mental contents.
3. Cognitive psychology is the study of cognition.
4. Cognitive science is an interdisciplinary field that extends the principles of
cognitive psychology to other systems that manipulate information.
Cognition and Brain-Non-invasive brain scanning allows correlations to be made between
human conscious experiences and patterns of brain activity. Studies of both visual and
auditoryperception allow distinctions to be made between brain regions that do and do not
show activity patterns that correlate with conscious experiences. Results from study of brain
lesions, application of drugs, and electromagnetic disruption of the function of specific brain
regions can be interpreted in combination with results from brain scans.
1. Neural correlates of the visual vertical meridian asymmetry by Taosheng Liu, David J. Heeger, and Marisa Carrasco in Journal of Vision (2007)
Volume 6: 1294–1306.
2. Jump up Hierarchical Processing of Auditory Objects in Humans by Sukhbinder Kumar, Klaas E Stephan, Jason D Warren, Karl J Friston and Timothy D Griffithsin in PLoS Comput
Biol. (2007) Volume 3:e100.
https://en.wikiversity.org/wiki/Cognition
What does cognition mean? (from Wikipedia)
79. Cognition enables individuals to understand their environment and paves
the way for the application of perception to explore and deepen the
understanding
The network of concepts and relationships enables the cognition and
perception needed to interpret daily experiences.
What does cognition mean? (from Wikipedia)
80. The system’s understanding of it’s worlds is inherently specified to the
form of the systems’ embodiment and is dependent on the system’s
history of interactions and it’s experiences.
So this property of making sense of it’s environmental interactions is one
of the foundations of a branch of cognitive science called “Enaction”
From https://mitpress.mit.edu/books/artificial-cognitive-systems
Emergent Paradigm of Cognitive Science
82. http://www.fil.ion.ucl.ac.uk/~karl/Whatever%20next.pdf, http://libertymotive.com/truth/
Top-Down Approach model emphasizes the use of background knowledge to predict content.
The information that needs to be communicated ‘upward’ is just the prediction error: the
divergence from the expected signal.
Top-down Expectation/Prediction
firm, not squishy
red and green, not just red
Prediction Error
fruit
stem
Bottom-up Observation
Andy Clark
83. Attempts to build an artificial cognitive system that can be positioned in a two-dimensional space, with one axis
defining a spectrum running from purely computational techniques to techniques strongly inspired by biological
models, and with another axis defining the level of abstraction of the biological model.
Adapted from https://mitpress.mit.edu/books/artificial-cognitive-systems.
Inspiration
Abstraction Level
Modelling decomposition of
hypothetical model of brain
Cognitive system modelled on
the microscopic organization of
the brain
Cognitive system based on
statistical learning of
specific domain rules
Cognitive system based on
artificial neural networks
Computational Biological
Low
High
IBM Brain Chip
Aspects of Modelling Cognitive Systems
85. Histogram of speed values
collected from June 1st 12:00 AM to June 2nd 12:00 AM
Histogram of travel time values
collected from June 1st 12:00 AM to June 2nd 12:00 AM
Traffic Data: First Peek
86. This distribution resembles a Gaussian Mixture
Model (GMM)
Multiple Gaussian Distributions: A Better Fit for Speed Observations?
87. Assume Normalcy to be uninterrupted traffic flow
July 2014 has no events so, we
hypothesize higher log-likelihood
score
June 2014 has many events so, we
hypothesize lower log-likelihood
score
-115655.8
(Closer to Normalcy)
-125974.3
88
Golden Gate Fields: Comparing Months with Varying Event Occurrences
88. Root Cause Analysis Action Recommendation
Find Triggers of Asthma
Derive the cause of asthma attacks
for a given patient using statistical
techniques + knowledge of asthma
and its triggers.
Minimize Asthma Attacks
Model actions based on the utility
theory (cost of actions & its rewards)
+ knowledge of action consequences.
Two Research Directions for kHealth Asthma with More Ddata…
90. • If an anomaly is detected on a link L and during time period [tst, tet], then the
anomaly is explained by an event if the event occurred in the vicinity within 0.5km
radius and during [tst-1, tet+1].
• CAVEAT: An anomaly may not be explained because of missing data.
Thanks to Dr. Krishnaprasad Thirunarayan for sharing this slide.
Spatio-temporal Co-occurrence Criteria
92. How can we get inspiration
from brain to computing?
93. One approach, recreate the brain in
silicon. A second approach, different
disciplines learn about the human brain,
get inspiration and learn things for
computing. (Slide on mapping from brain
to computing.)
94.
95. "The domain of cognitive science occupies the intersection of philosophy, neuroscience,
linguistics, cognitive psychology, and computer science (artificial intelligence)."
Gerrig, R. J., Zimbardo, P. G., Campbell, A. J., Cumming, S. R., & Wilkes, F. J. (2008). Psychology and life (Australian edition). Sydney: Pearson Education
Australia, p. 248.https://en.wikiversity.org/wiki/Cognitive_science.
Cognitive Science
96. https://mitpress.mit.edu/books/artificial-cognitive-systems
Two hallmarks of cognition:
● Prediction out of the past interactions.
● Learning new knowledge by making sense of its interactions with the
world around it.
The dependency on exploration and development is one of the reasons why
an artificial system requires a rich sensory-motor interface with its
environment and why embodiment plays such a pivotal role.
Hallmarks of Cognition
98. Reduced
CO level =>
better asthma
control
High CO influences
Wheezing Level (Low/High)
High CO High Luminosity
High Wheeze Low Luminosity
Low Wheeze
Carbon Monoxide
Wheeze
Luminosity
1Amit Sheth, Pramod Anantharam, Cory Henson, 'Physical-Cyber-Social Computing: An Early 21st Century Approach,' IEEE Intelligent Systems, vol. 28, no. 1, pp. 78-82,
Jan.-Feb., 2013. http://doi.ieeecomputersociety.org/10.1109/MIS.2013.20
Horizontal Operators
(Semantic Integration) operates
on data from heterogeneous
sources to create
Integrated or correlated
data streams.
Vertical Operators
(Semantic abstraction) operates on
artifacts at each level and transcends
them to the next level.
“a holistic treatment of data, information,
and knowledge from the PCS worlds to
integrate, correlate, interpret, and
provide contextually relevant abstractions to
humans.”1
Physical-Cyber-Social Computing for Actionable Insights from Multimodal Data
102. Brain GPS - Grid Cell/ Place Cells
● in 1971 John O’Keefe of University College London discovered “place cells” found in
Hippocampus.
● In 2005 May-Britt and Edvard Moser of Norwegian University of Science and Technology added a
new discovery : the existence of “grid cells” in the nearby cortex.
● The grid and place cells work together to constitute an internal navigation system.
● Grid cells provide a set of coordinates that enable a rat to navigate through its environment in
conjunction with other cells that recognize the positioning of the head and the borders of a room.
● Grid cell systems give you immediate sense of recognition of, yes, this is where I am. "That's my
home, there's my office, there's the entrance to the metro."
● Alzheimer's patients wander off into the night, as a consequence of the dying of neurons in the
entorhinal cortex and the hippocampus.
https://www.kth.se/blogs/prasanth/tag/india-and-sweden/ http://www.dallasnews.com/opinion/sunday-commentary/20130920-gps-may-reroute-the-brain.ece
http://www.scientificamerican.com/sciam/assets/media/multimedia/110514-MindListicle/1.jpg http://www.activebeat.com/health-news/nobel-prize-awarded-to-scientists-
who-discovered-brain-gps-system/ http://www.scientificamerican.com/article/10-big-ideas-in-10-years-of-brain-science/
103. http://www.scientificamerican.com/article/10-big-ideas-in-10-years-of-brain-science/
In the future optogenetics will allow us to decipher both how various
brain cells elicit feelings, thoughts and movements—as well as how
they can go awry to produce psychiatric disorders.
● Stanford scientists presented a technique for switching individual
neurons on or off with light in 2005.
● To probe how a certain class of neurons helps mice navigate
mazes, scientists insert electrodes into brain tissue and stimulate
thousands of neurons at a time.
● Scientists can tuck light-sensitive molecules into specific brain
cells to manipulate only those selected neuron types or networks.
● Shining a light makes those neurons either more or less active
and can elucidate their role in a behavior or disease.Mouse with optogenetic tools in operation, including implanted fiberoptic
and light-sensitive molecules produced in the brain, all representing
technologies developed in the Deisseroth lab at Stanford University by
graduate students Raag Airan, Feng Zhang, Ed Boyden, and Lief Fenno.
Credit: Raag Airan, Feng Zhang, Ed Boyden, and Lief Fenno
“Over the past decade hundreds of research groups have used optogenetics to learn how various networks of neurons
contribute to behavior, perception and cognition,” wrote Ed Boyden, a co-inventor of optogenetics.
Optogenetics
105. Philanthropist Paul Allen, co-founder of Microsoft
● In 2003 , Allen Institute of Brain Science began mapping gene activity in
mouse brain
● Genetic activity help researchers discover genes relevant to certain
diseases or behaviors.
● Institute launched a 10-year plan to examine where specific genes are
active and how these genetic circuits process the vast flow of
information into the brain.
● For example, the brain map showed that genes associated with autism
appear to be acting on a specific type of brain cell in neocortex. That
suggests "we should be looking at this particular type of cell in the
neocortex, and furthermore that we should probably be looking very
early in the prenatal stages for the origin of autism".
A top-down 3-D view of the cortico-connections
originating from multiple distinct cortical areas,
visualized as virtual tractography using Allen
Institute Brain Explorer software. Credit: Allen
Institute for Brain Science.
http://www.scientificamerican.com/sciam/assets/media/multimedia/110514-MindListicle/1.jpg
http://www.scientificamerican.com/article/10-big-ideas-in-10-years-of-brain-science/
Brain Mapping
106. Neurocognitive functions are cognitive functions linked with the functions of
particular areas, neural pathways, or cortical networks in the brain substrate layers
of neurological matrix at the cellular molecular level.
Their understanding is closely linked to the neuropsychology and cognitive
neuroscience, two disciplines that broadly seek to understand how the structure
and function of the brain relates to perception, de-fragmentation of concepts,
memory embed, association and recall both in the thought process and behavior.
https://en.wikipedia.org/wiki/Neurocognitive
Neurocognitive
107. The term 'cognitive neuroscience' was coined by George Miller and Michael Gazzaniga toward the end of the 1970s.
Cognitive neuroscience began to integrate the newly laid theoretical ground in cognitive science, that emerged
between the 1950s and 1960s.
Cognitive neuroscience is concerned with the scientific study of the biological processes and aspects that underlie
cognition, with a specific focus on the neural connections in the brain which are involved in mental processes.
It addresses the questions of how psychological/cognitive activities are affected or controlled by neural circuits in the
brain. Cognitive neuroscience relies upon theories in cognitive science coupled with evidence from neuropsychology,
and computational modeling.
Neurons play the most vital role, since the main point is to establish an understanding of cognition from a neural
perspective, along with the different lobes of the cerebral cortex.
David Marr (neuroscientist) concluded that one should understand any cognitive process at three levels of analysis:
• Computational.
• Algorithmic/representational.
• Physical levels of analysis.
https://en.wikipedia.org/wiki/Cognitive_neuroscience
Cognitive Neurocognitive