1. 1
A qualitative investigation of optical flow
algorithms for video denoising
Hannes Fassold, JOANNEUM RESEARCH
2022-11-28
2. Introduction & Motivation
• Optical flow = pixel-wise motion field between two images (e.g. video frames)
• Good optical flow estimation us crucial for many video processing tasks
• E.g. object tracking, denoising, super-resolution …
• Focus of work was to investigate how well optical flow algorithms perform
qualitatively when integrated into a state of the art video denoising algorithm.
• Both classic optical flow algorithms (e.g. TV-L1) as well as recent deep
learning based algorithm (like RAFT or BMBC) are taken into account.
• Tests are done with our own wavelet-based video denoising algorithm
• Is integrated (in a significantly extended version – safeguards etc.)
in the film restoration software “DIAMANT” by HS-Art.
• HS-Art was also the host for this (virtual) secondment
• Spin-off of JOANNEUM RESEARCH, located in Graz
2
3. Optical flow algorithm used in experiments
• The TV-L1 optical flow algorithm (GPU) is a very popular method for optical flow
estimation and employs variational calculus.
• The Dense Inverse Search (DIS) optical flow algorithm (CPU) is designed
specifically for runtime efficiency and integrated in OpenCV library.
• The NV optical flow algorithm is a very fast hardware-accelerated optical flow
provided on recent NVIDIA GPUs (Ampere are newer).
• The RAFT method is a recent deep learning based method
• One of best optical flow methods currently on the benchmark datasets.
• The BMBC algorithm is actually a frame interpolation algorithm, but it can be
employed also as an optical flow algorithm.
• The LIFE optical flow method is a neural network which is explicitly designed to
be robust against content with large motion or large brightness variations.
3
4. Experiment setup
• The 2-phase JR video denoiser is employed (with safeguards disabled)
• Phase 1 – wavelet-denoising using semi-local shrinkage functions
• Phase 2 – temporal fusion within a 3-frame sliding window
• In both phases, neighbor images are motion-compensated via optical flow
• We employ different optical flow algorithms (TV-L1, DIS, NV, RAFT, BMBC, LIFE)
for the motion compensation step within the denoiser
• Qualitative experiments are done with different kinds of content
• local motion only
• fast motion
• heavy noise
• strong flicker / brightness variations
4
6. Conclusions from qualitative investigation
• No clear ”winner” can be determined, at least for our application scenario
• Typical motion-compensation artifacts: halos, blurring, …
• Newer deep learning based optical flow algorithm (especially RAFT) are not
necessarily better than the classic methods like TV-L1 or NV
• Indicates also that there is a gap between quantitative measures of optical flow
performance (like endpoint error) on benchmark datasets and a qualitative
evaluation in a certain application context (e.g. for video denoising)
• There is a need for more realistic quantitative measures for the faithful
evaluation of optical flow algorithms
6
7. Acknowledgment
• This work was supported by European Union´s Horizon 2020 research and
innovation programme under grant number 951911 - AI4Media.
7
