1. Super Resolution для
сжатого видео
Моисейцев Алексей
Video Group
CS MSU Graphics & Media Lab
2. Only for
Maxus
Содержание
Введение
Example-based SR
SRME
HMRF SR
2
CS MSU Graphics & Media Lab (Video Group)
3. Only for
Maxus
Введение
3
CS MSU Graphics & Media Lab (Video Group)
4. Only for
Maxus
Введение
Lossless 2Mbps 1Mbps 256Kbps
Bi-cubic
source
Super-res
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 4
CS MSU Graphics & Media Lab (Video Group) 2004
5. Only for
Maxus
Введение
Алгоритмы SR:
Iterative Backprojection (IBP)
Projection Onto Convex Sets (POCS)
Probabilistic Methods
Maximum a posteriori (MAP)
Model-based approach (MBSR)
Example-based
6
CS MSU Graphics & Media Lab (Video Group)
6. Only for
Maxus
Введение
IBP POCS
C2 C1
C
f=Pg P1g
P2g
C3
P3g
g
7
CS MSU Graphics & Media Lab (Video Group)
7. Only for
Maxus
Введение
8
CS MSU Graphics & Media Lab (Video Group) Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience
and Remote Sensing Symposium, 2005
8. Only for
Maxus
Введение
Типичная скорость работы: 0.15-0.4 fps
9
CS MSU Graphics & Media Lab (Video Group)
9. Only for
Maxus
Содержание
Введение
Example-based SR
SRME
HMRF SR
10
CS MSU Graphics & Media Lab (Video Group)
10. Only for
Maxus
Example-based SR
В SR только по LR-кадрам
существует теоретический
предел качества
Иногда есть доступ и к
отдельным кадрам в
высоком разрешении
11
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
11. Only for
Maxus
Example-based SR
Registration
Для компенсации движения
используется алгоритм Lucas-Kanade:
12
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
12. Only for
Maxus
Example-based SR
Registration
В 99% случаев ошибка не превышает 0.15 LR-пикселя и 0.45 HR-пикселя
13
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
13. Only for
Maxus
Example-based SR
Построение пар HR-LR
Используются LR-блоки
8x8
Рассматриваются
повороты блоков
Ключ —
10 AC-коэффициентов LR-
блока и 68 граничных
пикселей HR-блока
14
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
14. Only for
Maxus
Example-based SR
Восстановление
15
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
15. Only for
Maxus
Example-based SR
Восстановление
16
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
16. Only for
Maxus
Example-based SR
Результат
source, frame 0 compressed frame 20 SR, frame 20
MJPEG, Q=50
Обучение на первом кадре
Восстановление двадцатого кадра
17
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
17. Only for
Maxus
Example-based SR
Результат
compressed frame 20 SR, frame 20 Spatial example-based SR,
frame 20
18
CS MSU Graphics & Media Lab (Video Group) Resolution enhancement of lowKlamer Schutte,using a high-resolution frame,
Tuan Phama, Lucas van Vlieta,
quality videos
SPIE 2006
18. Only for
Maxus
Key-frame based SR
Идея: использовать HR ключевые кадры
19
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008
19. Only for
Maxus
Key-frame based SR
Восстановление
20
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008
20. Only for
Maxus
Key-frame based SR
Результаты
21
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008
21. Only for
Maxus
Key-frame based SR
Результаты
22
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008
22. Only for
Maxus
Key-frame based SR
Результаты
23
CS MSU Graphics & Media Lab (Video Group) Super Resolution ofMukherjee, ISCASFrames, Fernanda Brandi, Ricardo de
Queiroz, Debargha
Video Using Key
2008
23. Only for
Maxus
Example-based SR
Слабая обоснованность метода
Требуется наличие специфичного
видеопотока
Зависимость от обучающей выборки
24
CS MSU Graphics & Media Lab (Video Group)
24. Only for
Maxus
Содержание
Введение
Example-based SR
SRME
HMRF SR
25
CS MSU Graphics & Media Lab (Video Group)
25. Only for
Maxus
SRME
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 26
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
26. Only for
Maxus
SRME
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 27
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
27. Only for
Maxus
SRME
— апостериорная
вероятность
— интересующее
решение
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 28
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
28. Only for
Maxus
SRME
Задание вероятностей (1)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 29
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
29. Only for
Maxus
SRME
Задание вероятностей (2)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 30
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
30. Only for
Maxus
SRME
Задание вероятностей (3)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 31
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
31. Only for
Maxus
SRME
Решение (1)
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 32
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
32. Only for
Maxus
SRME
Решение (2)
Две итерации:
Компенсация движения
Построение улучшенного кадра
Simultaneous motion estimation and resolution enhancement of
compressed low resolution video. 33
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
33. Only for
Maxus
SRME
source
Y-PSNRSimultaneous motion estimation and resolution enhancement of
= 20.44dB
compressed low resolution video. 34
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
34. Only for
Maxus
SRME
bilinear
Y-PSNRSimultaneous motion estimation and resolution enhancement of
= 21.98dB
compressed low resolution video. 35
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
35. Only for
Maxus
SRME
proposed SR
Y-PSNRSimultaneous motion estimation and resolution enhancement of
= 25.64dB
compressed low resolution video. 36
CS MSU Graphics & Media Lab (Video Group) Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
36. Only for
Maxus
Содержание
Введение
Example-based SR
SRME
HMRF SR
37
CS MSU Graphics & Media Lab (Video Group)
37. Only for
Maxus
HMRF SR
Standard decompression Smoothed image Sharpened image
Restoration of Compressed Video using Temporal Information, 38
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
38. Only for
Maxus
HMRF SR
Restoration of Compressed Video using Temporal Information, 39
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
39. Only for
Maxus
HMRF SR
Шум квантования (1)
Restoration of Compressed Video using Temporal Information, 40
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
40. Only for
Maxus
HMRF SR
Шум квантования (2)
Restoration of Compressed Video using Temporal Information, 41
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
41. Only for
Maxus
HMRF SR
Шум компенсации (1)
Restoration of Compressed Video using Temporal Information, 42
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
42. Only for
Maxus
HMRF SR
Шум компенсации (2)
— общее выражение
— независимость шума
Restoration of Compressed Video using Temporal Information, 43
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
43. Only for
Maxus
HMRF SR
Условие гладкости
Restoration of Compressed Video using Temporal Information, 44
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
44. Only for
Maxus
HMRF SR
— гладкость
— квантование
— компенсация
Restoration of Compressed Video using Temporal Information, 45
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
45. Only for
Maxus
HMRF SR
Решение
Градиент:
Restoration of Compressed Video using Temporal Information, 46
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
46. Only for
Maxus
HMRF SR
Решение
Упрощение:
Градиентный спуск:
Restoration of Compressed Video using Temporal Information, 47
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
47. Only for
Maxus
HMRF SR
Результаты
Restoration of Compressed Video using Temporal Information, 48
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
48. Only for
Maxus
HMRF SR
Результаты
Restoration of Compressed Video using Temporal Information, 49
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
49. Only for
Maxus
HMRF SR
Результаты
Restoration of Compressed Video using Temporal Information, 50
CS MSU Graphics & Media Lab (Video Group) Mark A. Robertson and Robert L. Stevenson, SPIE 2001
50. Only for
Maxus
Заключение
Рассмотрены Example-based и MAP методы
Многие методы используют информацию из
потока или требуют дополнительные
данные для работы
Прямые реализации алгоритмов сложны, но
тем не менее многие из них возможно
распараллелить (например, с
использованием CUDA)
51
CS MSU Graphics & Media Lab (Video Group)
51. Only for
Maxus
Литература
Simultaneous motion estimation and resolution enhancement of compressed low
resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
Restoration of Compressed Video using Temporal Information, Mark A.
Robertson and Robert L. Stevenson, SPIE 2001
DCT Quantization Noise in Compressed Images, Mark A. Robertson and Robert L.
Stevenson, 2004
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004
Z. Lin and H-Y. Shum. Fundamental limits of reconstruction-based
superresolution algorithms under local translation. PAMI, 2004
Resolution enhancement of low quality videos using a high-resolution frame,
Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and C.L. Luengo Hendriks.
Performance of optimal registration estimators, SPIE 2005
Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and
Remote Sensing Symposium, 2005
Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de
Queiroz, Debargha Mukherjee, ISCAS 2008 52
CS MSU Graphics & Media Lab (Video Group)
52. Only for
Maxus
Вопросы
?
53
CS MSU Graphics & Media Lab (Video Group)