[Partly in Japanese]
I present it in 関東コンピュータビジョン勉強会(2015/07/25).
Main References:
http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx
If you find a problem, please let me know.
Thanks!
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
[Paper introduction] GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
1. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
2015 / 7 / 26 (Fri.)
関東コンピュータビジョン勉強会
発表者: @hokkun_cv
GMMCP-Tracker:
Globally Optimal Generalized Maximum
Multi Clique Problem for Multiple Object Tracking
1
Afshin Dehghan, Shayan Modiri Assari, Mubarak Shah
University of Central Florida
2. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
About me
• 東大院・学際情報学府・M2
• 相澤研究室所属
• 普段は食べものの研究をしています
• 2014/5のCV勉強会(CNNについて)ぶりの発表
参加です
2
• Preferred Networksでインターン→アルバイト中
• メンターが@tabe2314さん
• 今日はその課題の中で出てきたタスクに関連する
論文を紹介します
3. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
対象とする問題
• Multiple Object Tracking (MOT)
• YouTubeデモ (GMMCP)
3
※筆者は物体追跡については専門ではないので細かいとこ
ろに誤りがある可能性があります.遠慮無く指摘をお願い
致します.
4. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
ちなみに
• 筆者らはMultiple Object Trackingにかかわる論文
をもうひとつCVPR2015で発表している(強い)
• Target Identity-aware Network Flow for Online Multiple
Target Tracking
• 筆頭著者も一緒(Ph.Dの学生,ちなみに去年も2本筆頭で発表.強い)
4
5. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
その他
• H. Possegger et al., In Defense of Color-based
Model-free Tracking
• モデルフリートラッキング(非detection based)
• T. Liu et al., Real-time part-based visual tracking
via adaptive correlation filters
• パートベースのトラッキング
• S. Tang et al., Subgraph Decomposition for Multi-
Target Tracking
5
6. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Naïvest)
6
Frame n Frame n+1
Bipartite
Matching
Problem
7. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Tracking
7
Detection
Data
Association
http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx
8. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Tracking
8
Detection
Data
Association
http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx
9. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Naïvest)
9
Frame n Frame n+1
Bipartite
Matching
Problem
10. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Naïvest)
10
Frame n Frame n+1
Bipartite
Matching
Problem
11. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Network Flow)
11
Frame n Frame n+1 Frame n+2 Frame n+3
sources
sinks
minimum-cost
maximum-flow
problem
• incorporating
motion feature
• multi-commodity
network
12. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
12
Frame
1
Frame
2
Frame
3
13. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
However,
• Data association with network flow is simplified
formulation of this problem
• Assuming no simplification is closer to the
tracking scenario in real world.
13
14. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Not Simplify)
14
Frame n
Frame n+1
Frame n+2
Frame n+3
重み
=
0.95
重み
=
0.10
うまいこと重みが最大
になるクリークを探す
15. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Preliminary: clique (クリーク)
• 任意の2点を結ぶ枝がある頂点集合のこと
• see wikipedia in detail
• 今回は「各クラスタから1つのノードを選んでで
きる部分グラフ」という理解でOK
15
16. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Not Simplify)
16
Frame n
Frame n+1
Frame n+2
Frame n+3
Input: k-partite complete
graph (完全k部グラフ)
A person form a clique
↓
maximum clique
problem
17. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
GMCP Tracker[1]
• The same team s ECCV 2012 paper
• They formulate MOT as generalized maximum
clique problem. (cf. former page)
17[1] Amir Roshan Zamir et al., GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs, ECCV, 2012.
18. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
However (2),
• Due to complexity of the model, these
approaches have been solved by approximate
solutions.
• GMCP Tracker also used a greedy local
neighborhood search, which is prone to local
minima.
• GMCP Tracker doesn t follow a joint optimization
for all the tracks simultaneously (one by one).
18
19. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Contribution
1. this approach doesn t involve any simplification
neither in formulation nor in optimization
(Binary Integer Problem).
2. they propose a more efficient occlusion
handling strategy, which can handle long-term
occlusions (e.g. 150 frames) and can speed-up
the whole algorithm.
19
20. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Contribution
1. this approach doesn t involve any simplification
neither in formulation nor in optimization
(Binary Integer Problem).
2. they propose a more efficient occlusion
handling strategy, which can handle long-term
occlusions (e.g. 150 frames) and can speed-up
the whole algorithm.
20
21. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
21
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans
22. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
22
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans
23. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 0: Low-level Tracklet
• In GMCP, the nodes at first step are each
detections.
23
Frames
1-‐10
• In GMMCP, the nodes are (low-level) tracklet
• How to find: bounding boxes that overlap more than
60% between two frames are regarded as being
connected.
24. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
24
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans
25. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 1: Mid-level Tracklet
25
• 各クラスタ(青円)からひとつのノード(赤線)
を選び,クリークを作る
Frames
1-‐10
Frames
11-‐20
Frames
21-‐30
Frames
31-‐40
Frames
41-‐50
Frames
51-‐60
26. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 1: Mid-level Tracklet
26
• エッジの重み = (見た目特徴) + (動き特徴)
• これを基に最適化をすると・・
Frames
1-‐10
Frames
11-‐20
Frames
21-‐30
Frames
31-‐40
Frames
41-‐50
Frames
51-‐60
27. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 1: Mid-level Tracklet
27
• このような三人の軌跡が同時に検出できる
• オクルージョンに対応するため,ダミーノードを
入れてある
Frames
1-‐10
Frames
11-‐20
Frames
21-‐30
Frames
31-‐40
Frames
41-‐50
Frames
51-‐60
28. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
28
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans
29. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 2: Final Trajectories
• The another but similar problem with step 1.
• They solve GMMCP:
• Nodes are Mid-level Tracklet
• For appearance feature, they use median (or average)
feature among detections in each frame
• For motion feature, they use middle point of mid-level
tracklet as the location of each node
29
30. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Appearance Affinity
• Feature: Invariant Color Histogram [2]
• Deformation and viewpoint invariant
• Affinity: Histogram Intersection
30[1] J. Domke et al., Deformation and Viewpoint Invariant Color Histogram, BMVC, 2006
min(H1[i], H2[i])
31. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Motion Affinity
31[1] J. Domke et al., Deformation and Viewpoint Invariant Color Histogram, BMVC, 2006
今の位置
前の位置+速度度から
予想される位置
32. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Optimization
• GMMCP is NP Hard, but they solve without any
simplification.
• They formulate GMMCP as Binary Integer
Problem (BIP, 0-1整数計画問題)
32
33. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
33http://www.dais.is.tohoku.ac.jp/ shioura/teaching/dais08/dais02.pdf
34. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
34http://www.dais.is.tohoku.ac.jp/ shioura/teaching/dais08/dais02.pdf
35. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Optimization
• GMMCP is NP Hard, but they solve without any
simplification.
• They formulate GMMCP as Binary Integer
Problem (BIP, 0-1整数計画問題)
35
• これは実は組合せ最適化と言われる問題
• cf. 0-1ナップザック問題,巡回セールスマン問
題
36. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
BIP in this case
• C is weight matrix (?)
• x is boolean column vector
• the elements of x is all of edges and nodes
• Ax = b is equality constraints
• Mx <= n is inequality constraints
36
37. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
37
各クラスタごとに1に
なってるのは定数K
Notation
: i th node in j th cluster
: edge between and h: Number of clusterseij
mn
vm
n
vi
j
vi
j
あるノードから伸び
るエッジはh-1(かゼ
ロ)
クリークを作ってい
るかどうか
3種の制約
38. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Contribution
1. this approach doesn t involve any simplification
neither in formulation nor in optimization
(Binary Integer Problem).
2. they propose a more efficient occlusion
handling strategy, which can handle long-term
occlusions (e.g. 150 frames) and can speed-up
the whole algorithm.
38
39. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
• Detector can detect not all the persons in each
frame
• Occlusion, Detection Error, …
• They add Dummy Node to each cluster
• Cost of dummy edge ( = edge connected to
dummy node) is fixed value.
39
40. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
40
= さっきまで出てきてた重み ( 見た目 + 動き )cj1
cj2 = 定数c_d
cj3 , cj4 = 0
41. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
• How/How many do we add dummy nodes?
• Many dummy nodes increase computational
complexity
• cf. case of GMCP:
• They add dummy node by the motion-based way
• ある答えに対して等速度運動を仮定して,大きくハズレ
てしまうようなクラスタにダミーノードを足す
• Many dummy nodes increase computational
complexity (大事なので2度)
41
42. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
• Aggregated Dummy Nodes (ADN)
• no longer be boolean variable
• can take any integer value
• add only one ADN to each cluster
• Not connected to other nodes!
• New Solution: Mixed-Binary-Integer Programming
42
Constraint 1 Constraint 2 Constraint 3
各クラスタごとに1に
なってるのは定数K
あるクラスタから伸
びるエッジは1か0
クリークを作ってい
るかどうか
43. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
43
cj1
cj2 はない (
cj3 , cj4= 0
cd
2
cj3 , cj4 =
= さっきまで出てきてた重み ( 見た目 + 動き )
51.
GMMCP
Tracker:
Globally
Op3mal
Generalized
Maximum
Mul3
Clique
Problem
for
Mul3ple
Object
Tracking
TUD-Stadmitte
Mid-‐level
Tracklets
Final
Trajectories
52.
GMMCP
Tracker:
Globally
Op3mal
Generalized
Maximum
Mul3
Clique
Problem
for
Mul3ple
Object
Tracking
53.
GMMCP
Tracker:
Globally
Op3mal
Generalized
Maximum
Mul3
Clique
Problem
for
Mul3ple
Object
Tracking
54. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
まとめ (拝借)
• Formulate MOT as GMMCP
• a new graph theoretic problem
• Formulate GMMCP as a MBIP
• GMMCP is NP Hard but no approximate solutions
• An efficient occlusion handling through AND
• Performance close to real-time
• Improving state-of-art on several sequences
55. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
project page
• 本物のproject pageが情報量多くてこれがCVPR複
数本2年連続で通す人のページか,と思いました
• http://crcv.ucf.edu/projects/GMMCP-Tracker/
55
56. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Implementation Detail
• Detection: DPM
• K: target-specific
• (1st layer ) number of cluster: 5 (2-6で実験)
• (2nd layer) number of cluster: 6
56
Notas del editor
Data AssociationベースのトラッカーはとてもDetectorの性能に依存するよね
だからどっちも一緒に勉強しようね
この手のはいくつかはある
ベクトルは黒板でsつめい
----- Meeting Notes (5/4/15 14:46) -----
fix the video