Geometry based block partitioning （GBP） has been shown to achieve better performance than the tree structure based block partitioning （TSBP） of H.264. However, the residual blocks of GBP mode after motion compensation still present some non-vertical/non-horizontal orientations, and the conventional discrete cosine transform （DCT） may generate many high-frequency coefficients. To solve this problem, in this paper we propose a video coding approach by using GBP and reordering DCT （RDCT） techniques. In the proposed approach, GBP is first applied to partition the macroblocks. Then, before performing DCT, a reordering operation is used to adjust the pixel positions of the residual macroblocks based on the partition information. In this way, the partition information of GBP can be used to represent the reordering information of RDCT, and the bitrate can be reduced. Experimental results show that, compared to H.264/AVC, the proposed method achieves on average 6.38% and 5.69% bitrate reductions at low and high bitrates, respectively.
