SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-resolution

Zhengxue Wang Zhiqiang Yan^* Jian Yang^*

^*corresponding author
PCA Lab, Nanjing University of Science and Technology, China

⭐ Amazing Depth Super-Resolution Reconstruction ⭐

SGNet introduces a novel perspective that exploits the gradient and frequency domains for the structure enhancement of DSR task. Here are some typical super-resolution cases.

Abstract

Depth super-resolution (DSR) aims to restore high-resolution (HR) depth from low-resolution (LR) one, where RGB image is often used to promote this task. Recent image guided DSR approaches mainly focus on spatial domain to rebuild depth structure. However, since the structure of LR depth is usually blurry, only considering spatial domain is not very sufficient to acquire satisfactory results. In this paper, we propose structure guided network (SGNet), a method that pays more attention to gradient and frequency domains, both of which have the inherent ability to capture high-frequency structure. Specifically, we first introduce the gradient calibration module (GCM), which employs the accurate gradient prior of RGB to sharpen the LR depth structure. Then we present the Frequency Awareness Module (FAM) that recursively conducts multiple spectrum differencing blocks (SDB), each of which propagates the precise high-frequency components of RGB into the LR depth. Extensive experimental results on both real and synthetic datasets demonstrate the superiority of our SGNet, reaching the state-of-the-art.

Method

Overview of our Structure Guided Network (SGNet). Given I_rgb and D_lr^up as input, the Gradient Calibration Module (GCM) first maps them into gradient domain, producing F_ge with sharp depth structure. Then, I_rgb, D_lr and F_ge are fed into the Frequency Awareness Module (FAM) to estimate frequency enhanced depth feature D_fe via recursive Spectrum Differencing Blocks (SDB). ↑: bicubic up-sample. Grad. Mapping: Gradient Mapping. Freq. Mapping: Frequency Mapping.

Spectrum differencing block (SDB). Green dashed box: 1x1 convolution. Gray rectangular box: a 1x1 convolution and an invertible neural network.

Quantitative Comparison

Visual Comparison

Visual results and error maps on NYU-v2 dataset (x16).

Visual results and error maps on RGB-D-D dataset (x16).

Visual results on real-world RGB-D-D dataset.

BibTex

                    
@article{wang2023sgnet,
  title={SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-Resolution},
  author={Wang, Zhengxu and Yan, Zhiqiang and Yang, Jian},
  journal={arXiv preprint arXiv:2312.05799},
  year={2023}
}

Contact

For any questions, please contact {zxwang,yanzq}@njust.edu.cn