Speaker: dr. Rick Quax Date:13-11-2015 Abstract: Quantifying synergy among stochastic variables is an important open problem in information theory. Information synergy occurs when multiple sources together predict an outcome variable better than the sum of single-source predictions. It is an essential phenomenon in biology such as in neuronal networks and cellular regulatory processes, where different information flows integrate to produce a single response, as well as in statistical inference tasks such as feature selection in machine learning. Here we propose a metric of synergistic entropy and synergistic information from first principles. The proposed measure relies on so-called synergistic stochastic variables (SRVs) which are constructed to have zero mutual information about individual source variables but non-zero mutual information about the complete set of source variables. We prove several basic and desired properties of our measure, including bounds and additivity properties. In addition we prove several important consequences of our measure, including the fact that different types of synergistic information may co-exist between the same sets of variables.Our measure may be a marked step forward in the study of multivariate information theory and its numerous applications.

1. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Quantifying information synergy
Using intermediate variables

2. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Information integration
or

3. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Setting
1X
Y2X
3X
 p X  p Y X
 1 2, ,...X X X How much synergy in Y about X
…

4. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
INFORMATION THEORY
Basics of

5. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Entropy of a coin flip
0 ( 0) 0.5
1 ( 1) 0.5
p X
p X
 

 
Carries 1 bit of information
0 ( 0) 0
1 ( 1) 1
p X
p X
 

 
Carries 0 bits of information
X 
X 

6. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Entropy of a coin flip
In general:
 
2
0,1
1
( ) ( ) log
( )x
H X p X x
p X x
  


0 ( 0) 0.5
1 ( 1) 0.5
p X
p X
 

 
Carries 1 bit of information
0 ( 0) 0
1 ( 1) 1
p X
p X
 

 
Carries 0 bits of information
X 
X 

7. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Mutual information between coins
( 0) 0.5
( 1) 0.5
p X
p X
 

 
Transform
X Y
( | )p Y X ( | )p X Y ?

8. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
( 0) 0.5
( 1) 0.5
p X
p X
 

 
Transform
X Y
( | )p Y X ( | )p X Y
( | ) 1p Y x X x   1 bit transferred
( | ) 1/ 2p Y x X x   0 bits transferred
?
eq( | )p Y x X x p  
 
,
( , )
: ( , )log
( ) ( )x y
p x y
I X Y p x y
p x p y
 
Mutual information between coins

9. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Summary of information theory
  ( )log ( )
X x
H X p x p x

  “Entropy”
“Mutual information” 
   
,
( , )
: ( , )log ,
( ) ( )
H .
x y
p x y
I X Y p x y
p x p y
H X X Y

 


10. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
SYNERGISTIC INFORMATION
What is

11. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Individual versus synergistic
X1 X2 Y
0 0 0
0 1 0
1 0 1
1 1 1
X1 X2 Y
0 0 0
0 1 1
1 0 1
1 1 0
A
B
A
100% synergistic100% individualistic

12. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Individual versus synergistic
X1 X2 Y
0 0 0
0 1 0
1 0 1
1 1 1
X1 X2 Y
0 0 0
0 1 1
1 0 1
1 1 0
A
B
A
100% synergistic100% individualistic
?

13. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Naive attempt
1X
Y2X
3X
 p X  p Y X
“Whole-Minus-Sum” (WMS)
   "synergy" : :
WMS
i
i
I X Y I X Y  

14. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Naive attempt
1X
Y2X
3X
 p X  p Y X
“Whole-Minus-Sum” (WMS)
   "synergy" : :
WMS
i
i
I X Y I X Y  

15. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Previous attempts
Candidate redundancy
measure
Achilles’ heel
WholeMinusSum (WMS) Redundant XOR
Imin Copy two bits
IΛ Noisy output
Iα Correlated inputs AND
… …

16. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Terminology
Stochastic variable
iX
 i i
X X
X Y
Set of variables
 Pr |Y X

17. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
HOW MUCH SYNERGISTIC INFORMATION
DOES Y CONTAIN ABOUT X
New theory to quantify

18. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Definition: synergistic
 
 
is synergystic about iff
: 0,
: 0.
j
j
j i
S X
I S X
I S X


 j j
S S

19. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
First intuition
( : ) synergyI Y S 

20. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Counterexample
 1 2,X X X
 0,1,2iX 
Pr( ) 1 9X 
 1 1 2 mod3S X 
 2 12 1 mod3S X 
 1 1: 0I S X 
 2 1: 0I S X 
 1 2 1, : 1.22I S S X 
Y=X1 would be synergistic?!

21. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Second intuition
( : ) synergyj
j
I Y S 

22. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Counterexample
 1 2 3, ,X X X X
 0,1iX 
 Pr X
1 1 2S X X 
2 2 3S X X 
3 1 3S X X 
4 1 2 3S X X X  
Choosing Y={S1,S2} would result in 3 bits of synergy

23. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Orthogonal decomposition
   
 
   
: , such that
, : ,
: 0,
: : .
D B B B
I B B B H B
I B A
I B A I B A






P
P
P
a
A B
B
BP
,

24. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Final result
  *
1 1: ; ,..., ,i i i i i
S S S D S S S 
 
   * *syn ordering for
max : .
i
iS S S
I X Y I Y S
  
For a given ordering of Si

25. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Proved consequences
   syn, :X Y I X Y H S   
   synI X Y H S No ‘overcounting’
No ‘undercounting’
 3: not counted already countedI Y S 

26. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Further proved consequences
 syn 0.I X Y 
   syn : .I X Y I X Y 
         syn syn ' '' '
.i i i ii j i i
I X Y I X Y  
 syn 1 0.I X Y 
 syn 1 0.I X X 
         syn max.i i
S
I X X H X H X 
   
     1( : ) ,..., max , where .N i iI X Y H X X H X Y X  

27. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Prevailing framework

Partial Information Decomposition (PID)
  indiv syn:I X Y I I 
Williams, Paul L., and Randall D. Beer. "Nonnegative
decomposition of multivariate information." arXiv
preprint arXiv:1004.2515 (2010).

28. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Prevailing framework

Partial Information Decomposition (PID)
  indiv syn:I X Y I I 
Williams, Paul L., and Randall D. Beer. "Nonnegative
decomposition of multivariate information." arXiv
preprint arXiv:1004.2515 (2010).

29. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Operational meaning of synergy
Susceptibilitytoperturbing
Random Y Synergistic Y
†
1X
Y2X
3X
†
1 1X X

30. Rick Quax: Computational Science, University of Amsterdam, The Netherlands.
Conclusion
• New framework to think about synergy
• Start only from two simple ingredients
– Definition of ‘synergistic’
– Correlate Y with intermediate S instead of X
• (=incompatible with PID)
• Limitation: computationally expensive
  indiv syn:I X Y I I 
https://bitbucket.org/rquax/info_metricsSoftware: