2020/02/28 Deep Learning JP: http://deeplearning.jp/seminar-2/

1. DEEP LEARNING JP
[DL Papers]
http://deeplearning.jp/

4. Mi = (S, A, Ri, Ti, Ti,0, r, H)
S A R T
T0 γ H

5. Mi ∼ p(M) Mi p(M)
Mi Ri Ti
p(M)
i = Ntrain
i = 2
i = 1
Mi =
(S, A, Ri, Ti, Ti,0, r, H)
i = Ntest
i = 2
i = 1
Mi =
(S, A, Ri, Ti, Ti,0, r, H)
∼i.i.d. p(M)

6. T R

7. bt = p(R, T|τ:t), where τ:t = {s0, a0, r1, s1, a1, . . . , st}
bt
s+
t = [st, bt] s+
t

8. bt = p(R, T|τ:t)

10. bt = p(R, T|τ:t)
pθ(m|τ:t) bt
bt = pθ(m|τ:t)
pθ(m|τ:t)
p(M)
πψ(at |s+
t )
R, T
bt
bt

11. πψ(at |s+
t = {st, bt})
bt = p(R, T|τ:t)
ψ R, T ∼i.i.d. p(R, T)
bt = p(R, T|τ:t) πψ(at |s+
t = {st, bt})
p(R, T|τ:t)

12. M+
= (S+
, A, R+
, T+
, T+
0 , r, H)
R+
T+
st ∈ S+
at ∈ At rt
R+
T+
R+
T+
p(M)
πψ(at |s+
t )
πψ(at |s+
t )

13. bt = p(R, T|τ:t)
m pθ(τ:H+ |a:H+−1)
bt
{
Ep(M,τ:H+)[log pθ(τ:H+ |a:H+−1)]
= Ep(M,τ:H+)[∑ log Epθ(m|τt)pθ(st+1 |at, st, m)pθ(rt+1 |st, at, m)]
pθ(m|τ:t) = p(R, T|τ:t) = bt
pθ(m|τ:t) st st+1
at
rt+1
m

14. m H+

15. bt = p(R, T|τ:t)
pθ(m|τ:t) bt
bt = pθ(m|τ:t)
pθ(m|τ:t)
p(M)
πψ(at |s+
t )
R, T
bt
bt

16. st

17. q(m|τ:t)

22. ̂bt
̂bt
̂bt