High Efficiency Video Coding (HEVC) improves the encoding efficiency by utilizing sophisticated tools such as flexible Coding Tree Unit (CTU) partitioning. The Coding Unit (CU) can be split recursively into four equally sized CUs ranging from 64×64 to 8×8 pixels. At each depth level (or CU size), intra prediction via exhaustive mode search was exploited in HEVC to improve the encoding efficiency and result in a very high encoding time complexity. This paper proposes an Intra CU Depth Prediction (INCEPT) algorithm, which limits Rate-Distortion Optimization (RDO) for each CTU in HEVC by utilizing the spatial correlation with the neighboring CTUs, which is computed using a DCT energy-based feature. Thus, INCEPT reduces the number of candidate CU sizes required to be considered for each CTU in HEVC intra coding. Experimental results show that the INCEPT algorithm achieves a better trade-off between the encoding efficiency and encoding time saving (i.e., BDR/∆T) than the benchmark algorithms. While BDR/∆T is 12.35% and 9.03% for the benchmark algorithms, it is 5.49% for the proposed algorithm. As a result, INCEPT achieves a 23.34% reduction in encoding time on average while incurring only a 1.67% increase in bit rate than the original coding in the x265 HEVC open-source encoder.

1. INCEPT: Intra CU Depth Prediction for HEVC
Vignesh V Menon, Hadi Amirpour, Christian Timmerer and Mohammad Ghanbari
Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), University of Klagenfurt, Austria
06 October 2021
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 1

2. Outline
1 Introduction
2 INCEPT Algorithm
3 Evaluation
4 Conclusions and Future Directions
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 2

3. Introduction
Introduction
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 3

4. Introduction
Introduction
Background of High Efficiency Video Coding (HEVC)3
The compression efficiency of HEVC has been improved greatly compared to AVC1 with
the adoption of numerous advanced tools.
HEVC has a flexible quad-tree coding block partitioning structure and uses the Coding
Units (CUs) and Prediction Units (PUs) concept to manage the partition.2
Each video frame is divided into non-overlapping predefined size Coding Tree Unit (CTUs),
and each CTU is then recursively divided into CUs. A CU can be further partitioned into
PUs and Transform Units (TUs).
The partitioning is recursive within a quad-tree hierarchy making the codec computation-
ally intensive.
1
T. Wiegand et al. “Overview of the H.264/AVC video coding standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7 (2003),
pp. 560–576.
2
Ekrem Çetinkaya et al. “CTU depth decision algorithms for HEVC: A survey”. In: Signal Processing: Image Communication 99 (2021), p. 116442. issn:
0923-5965. doi: https://doi.org/10.1016/j.image.2021.116442. url: https://www.sciencedirect.com/science/article/pii/S0923596521002113.
3
G. J. Sullivan et al. “Overview of the high efficiency video coding (HEVC) standard”. In: IEEE Transactions on circuits and systems for video technology
22.12 (2012), pp. 1649–1668.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 4

5. Introduction
Introduction
CU partitioning in HEVC
32x32
16x16
8x8
depth 1
depth 0
depth 2
depth 3
Figure: An example of the CU partitioning structure of a CTU and its corresponding quad-tree
structure. The white nodes represent CUs that have been partitioned.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 5

6. Introduction
Introduction
Intra Coding of High Efficiency Video Coding (HEVC)6
Intra frames are essential to conditions such as mobile devices with limited computational
power, transmission over error-prone channels, and frequent random access.4
In intra coding, PU’s size is generally equal to the corresponding CU. In addition, up to 35
prediction modes, including DC mode, Planar mode, and 33 angular modes, are provided
for intra prediction in HEVC, enabling more accurate predictions.
The increase in CU depth and prediction modes also causes higher coding complexity than
Advanced Video Coding (AVC).5
4
Yun Zhang et al. “Statistical Early Termination and Early Skip Models for Fast Mode Decision in HEVC INTRA Coding”. In: ACM Trans. Multimedia
Comput. Commun. Appl. 15.3 (July 2019). issn: 1551-6857. doi: 10.1145/3321510. url: https://doi.org/10.1145/3321510.
5
Wiegand et al., “Overview of the H.264/AVC video coding standard”.
6
Sullivan et al., “Overview of the high efficiency video coding (HEVC) standard”.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 6

7. Introduction
Intra CU Depth Estimation
Start CTU
Depth i ∈
[dmin, dmax]
No
d > dmax
End CTU
PU mode decisions
i = i + 1 for next
CU
No
Yes
Yes
Figure: Quad-tree CU algorithm for partitioning of a CTU.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 7

8. INCEPT Algorithm
INCEPT Algorithm
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 8

9. INCEPT Algorithm Phase 1: Feature Extraction
INCEPT Algorithm
Phase 1: Feature Extraction
Compute texture energy per Coding Tree Unit (CTU)
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
EDCT =
w
X
i=1
h
X
j=1
e|( ij
wh
)2−1|
|DCT(i − 1, j − 1)| (1)
where w and h are the width and height of the block, and DCT(i, j) is the (i, j)th DCT
component when i + j > 2, and 0 otherwise.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 9

10. INCEPT Algorithm Phase 1: Feature Extraction
INCEPT Algorithm
Phase 2: CU Depth Prediction
For each CTU:
Inputs:
HL, HA, HAL : weighted DCT energy of the neighboring CTUs
mL, mA, mAL : mean of the CU depths of the neighboring CTUs
vL, vA, vAL : variance of the CU depths of the neighboring CTUs
Output: dmin and dmax
Step 1: Compute Hmin and Hmax , mmin and mmax and vmin and vmax
Hmin = min(HL, HA, HAL)
Hmax = max(HL, HA, HAL)
mmin = min(mL, mA, mAL)
mmax = max(mL, mA, mAL)
vmin = min(vL, vA, vAL)
vmax = max(vL, vA, vAL)
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 10

11. INCEPT Algorithm Phase 1: Feature Extraction
INCEPT Algorithm
Phase 2: CU Depth Prediction
Step 2: Compute HC .
Step 3: Determine dmin and dmax
if HC < Hmin then
dmax = dmmin + vmin
2 + φe
else
dmax = dmmax + vmax
2 + φe
if HC > Hmax then
dmin = bmmax − vmin
2 + ψc
else
dmin = bmmin − vmax
2 + ψc
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 11

12. INCEPT Algorithm Phase 1: Feature Extraction
INCEPT Algorithm
Depth prediction accuracy
Table: Depth prediction accuracy of INCEPT algorithm over JVET sequences and QPs.
Video QP22 QP26 QP30 QP34 Average
CatRobot 97.93% 95.98% 95.34% 94.76% 96.00%
DaylightRoad2 98.04% 97.86% 97.02% 96.39% 97.33%
FoodMarket4 99.32% 99.1% 98.79% 97.34% 98.64%
ParkRunning3 96.89% 95.34% 94.71% 94.03% 95.24%
Average 98.05% 97.07% 96.46% 95.63% 96.80%
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 12

13. Evaluation
Evaluation
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 13

14. Evaluation
Evaluation
Test Methodology
Test videos: (i) JVET test sequences,7 (i) MCML test sequences,8 and (ii) SJTU test
sequences9 representing various types of contents.
System: Dual-processor server with Intel Xeon Gold 5218R (80 cores, 2.10 GHz)
The presented algorithms were implemented in x265 v3.410 and tested with the veryslow
preset.
Sequences were encoded with ALL intra configuration and QPs ∀{22, 26, 30, 34}.
The lower resolution sources were generated from the original video source by applying
bi-cubic scaling using FFmpeg.11
7
Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018.
8
Manri Cheon and Jong-Seok Lee. “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience”. In: IEEE
Transactions on Circuits and Systems for Video Technology 28.7 (2018), pp. 1467–1480. doi: 10.1109/TCSVT.2017.2683504.
9
L. Song et al. “The SJTU 4K Video Sequence Dataset”. In: Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013) (July 2013).
10
MulticoreWare Inc. x265 HEVC Encoder/H.265 Video Codec. url: http://x265.org/.
11
FFmpeg. FFmpeg Documentation. url: https://ffmpeg.org/ffmpeg.html.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 14

15. Evaluation
Evaluation
Test Methodology
Metrics:
∆T: the cumulative time savings for all bitrate representations compared with the stand-
alone encoding
Bjøntegaard delta rates,12 BDRP and BDRV : average increase in bitrate of the represen-
tations compared with that of the stand-alone encoding to maintain the same PSNR and
VMAF.
BDRP
∆T and BDRV
∆T are calculated to compare the performance of the algorithms. The lower
is the value of BDR
∆T , the better is the performance of the algorithm.
12
G. Bjontegaard. “Calculation of average PSNR differences between RD-curves”. In: VCEG-M33 (2001).
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 15

16. Evaluation
Evaluation
Experimental Results
Table: ∆T and BDR comparison between the INCEPT algorithm and the benchmark algorithms.
ADTS13 SCDP14 INCEPT
Video ∆T BDRP BDRV ∆T BDRP BDRV ∆T BDRP BDRV
CatRobot 13.74% 2.36% 2.02% 24.97% 3.89% 3.71% 24.75% 3.08% 3.25%
DaylightRoad 16.38% 1.25% 1.19% 26.30% 3.15% 2.44% 26.20% 1.72% 1.54%
FoodMarket 16.15% 1.06% 1.12% 19.00% 2.56% 1.26% 20.09% 1.40% 0.72%
Basketball 13.75% 1.96% 1.68% 18.69% 4.82% 3.29% 19.13% 2.16% 1.88%
Bunny 15.40% 1.98% 2.03% 18.11% 3.32% 3.09% 18.69% 1.67% 1.69%
Lake 13.01% 1.08% 0.97% 22.54% 2.95% 2.12% 22.89% 1.19% -2.25%
BundNightScape 16.68% 1.08% 0.99% 28.18% 2.95% 4.10% 28.02% 1.43% 1.73%
CampfireParty 12.27% 0.78% 1.12% 22.11% 1.88% 2.64% 23.41% 0.82% 1.36%
Fountains 17.61% 0.88% 1.07% 25.12% 2.72% 2.66% 26.90% 1.53% 1.61%
Average 15.00% 1.38% 1.35% 22.78% 3.14% 2.81% 23.34% 1.67% 1.28%
13
Xin Lu, Chang Yu, and Xuesong Jin. “A fast HEVC intra-coding algorithm based on texture homogeneity and spatio-temporal correlation”. In: EURASIP
Journal on Advances in Signal Processing 37 (2018). doi: https://doi.org/10.1186/s13634-018-0558-4.
14
Zhang et al., “Statistical Early Termination and Early Skip Models for Fast Mode Decision in HEVC INTRA Coding”.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 16

17. Evaluation
Evaluation
Experimental Results
SCDP ADTS INCEPT
0
2
4
6
8
10
12
14
BDRv
/
T
12.35%
9.03%
5.49%
Figure: Comparison of BDRV
∆T with benchmark algorithms
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 17

18. Evaluation
Evaluation
Experimental Results
QP22 QP26 QP30 QP34
0
5
10
15
20
25
30
35
T
(%)
28.60%
25.54%
23.34%
21.56%
Figure: Average time saving using INCEPT algorithm for various QPs
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 18

19. Conclusions and Future Directions
Conclusions and Future Directions
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 19

20. Conclusions and Future Directions
Conclusions
We proposed fast intra CU depth prediction algorithm for HEVC encoding.
We analyzed the algorithm by comparing against two benchmark algorithms after integrat-
ing them into the x265 open-source HEVC encoder.
Experimental results demonstrate that the proposed INCEPT algorithm decreased the over-
all encoding time by 23.34% with a negligible increase in bitrate.
BDR
∆T metric is better for INCEPT compared to the two benchmark algorithms.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 20

21. Conclusions and Future Directions
Future Directions
More encoding time can be saved by reducing the time taken for PU mode decisions for
each CU15,16,17.
The proposed INCEPT algorithm can be easily extended for the VVC standard18.
15
Lu, Yu, and Jin, “A fast HEVC intra-coding algorithm based on texture homogeneity and spatio-temporal correlation”.
16
Jinzheng Lu and Yixian Li. “Fast Algorithm for CU Partitioning and Mode Selection in HEVC Intra Prediction”. In: 2019 12th International Congress on
Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). 2019, pp. 1–5. doi: 10.1109/CISP-BMEI48845.2019.8966035.
17
Tao Zhang et al. “Fast Intra-Mode and CU Size Decision for HEVC”. In: IEEE Transactions on Circuits and Systems for Video Technology 27.8 (2017),
pp. 1714–1726. doi: 10.1109/TCSVT.2016.2556518.
18
Gary Sullivan. “Versatile Video Coding (VVC) Arrives”. In: 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP).
2020, pp. 1–1. doi: 10.1109/VCIP49819.2020.9301847.
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 21

22. Conclusions and Future Directions
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Hadi Amirpour (hadi.amirpourazarian@aau.at)
Christian Timmerer (Christian.Timmerer@aau.at)
Mohammad Ghanbari (ghan@essex.ac.uk)
Vignesh V Menon INCEPT: Intra CU Depth Prediction for HEVC 22