随着深度学习的发展，单图像超分辨率技术取得了长足的进步。然而，现有的大多数研究都专注于卷积神经网络来构建具有大量层数的更深层次的网络模型。这些方法难以应用于现实场景，因为它们不可避免的伴随着复杂操作所带来的计算和内存成本问题。为此，我们提出了一种用于单图像超分辨率重建的轻量级混合模型——轻量级融合CNN-Swin Transformer网络。具体来说，我们使用带有移动窗口的Swin Transformer块充分学习图像的长期依赖性，并构建了一个基于CNN的局部特征提取块来有效地提取图像的局部特征细节。同时，设计了一个多路径动态卷积块来学习图像的边缘特征。实验结果表明，与基于Transformer的单图像超分辨率模型相比，本文提出的模型取得了更好的结果。
With the development of deep learning, single image super-resolution technology has made great progress. However, most of the existing research focuses on convolutional neural networks to construct deeper network models with a large number of layers. These methods are difficult to apply to real-world scenarios because they inevitably come with computational and memory costs associated with complex operations. Therefore, we propose a lightweight hybrid model for super-resolution reconstruction of single image—lightweight fusion CNN-Swin Transformer network. Specifically, we use Swin Transformer block with shifted windows to fully learn the long-term dependence of the image, and build a CNN-based local feature extraction block to effectively extract the local feature details of the image. Meanwhile, a multipath dynamic convolution block is designed to learn the edge features of the image. Experimental results show that compared with the single image super-resolution model based on Transformer, the proposed model achieves better results.
With the development of deep learning, single image super-resolution technology has made great progress. However, most of the existing research focuses on convolutional neural networks to construct deeper network models with a large number of layers. These methods are difficult to apply to real-world scenarios because they inevitably come with computational and memory costs associated with complex operations. Therefore, we propose a lightweight hybrid model for super-resolution reconstruction of single image—lightweight fusion CNN-Swin Transformer network. Specifically, we use Swin Transformer block with shifted windows to fully learn the long-term dependence of the image, and build a CNN-based local feature extraction block to effectively extract the local feature details of the image. Meanwhile, a multipath dynamic convolution block is designed to learn the edge features of the image. Experimental results show that compared with the single image super-resolution model based on Transformer, the proposed model achieves better results.
如图1所示，LFCSTN主要有四个部分组成：浅层特征提取、轻量级融合CNN-Swin Transformer模块(LFCSTM)、多路径动态卷积块(MDCB)和图像重建。我们将 I L R 和 I S R 分别定义为输入LR图像和重建SR图像。首先，我们假设给定一个低分辨率图像 I L R ∈ R H × W × C ，其中H和W分别为LR图像的高度和宽度，C表示特征通道数。我们使用3 × 3卷积层从 I L R 中提取浅层特征 F s f ∈ R H × W × C 为
F s f = f c o n v 3 ( I L R ) (1)
其中， f c o n v 3 ( ⋅ ) 表示卷积层， F s f 表示提取的浅层特征。然后将提取出来的浅层特征分别作为LFCSTM和MDCB输入
F n = f L n ( f L n − 1 ( ⋯ ( f L 1 ( F s f ) ) ) ) (2)
F e f = f M ( F s f ) (3)
其中， f L n 表示第n个LFCSTM模块的映射， F n 表示第n个LFCSTM的输出， f M 表示MDCB的映射， F e f 表示MDCB提取到的边缘特征。然后，我们将 F n 输入到一个卷积组中，进一步提取图像的深层特征
F d f = f C G ( F n ) (4)
其中， F d f 表示提取的深层特征， f C G ( ⋅ ) 表示用于提取图像深层特征的卷积组，主要由3 × 3卷积层和LeakyReLU激活函数组成。最后将 F d f 和 F e f 同时输入图像重建模块得到SR图像 I S R ∈ R H × W × C 。
I S R = f p ( F d f + F e f ) (5)
其中 f p ( ⋅ ) 代表亚像素卷积层。关于LFCSTM和MDCB更详细的部分，我们将在之后的小节进行介绍。
3.2. Swin Transformer块(STB)
STB作为LFCSTM中的Transformer分支部分，结构如图2(b)所示。Swin Transformer的构建方法是将 Transformer 块中的标准多头自注意力(Multi-Head Self-Attention, MSA) [8] 模块替换为基于移动窗口的模块，其它层保持不变。首先，我们假设给定大小为 H × W × C 的浅层特征 F s f 作为STB的输入，Swin Transformer
将输入重塑为 H W M 2 × M 2 × C 的特征，方法是将输入划分为不重叠的 M × M 的局部窗口，其中 H W M 2 是窗口的总数。然后，再为每个窗口分别计算局部注意力，得到局部窗口特征 F l w f ∈ R M 2 × C 。对于局部窗口特征 F l w f ，
查询、键和值矩阵：Q、K和V的计算公式为
Q = F l w f P Q (7)
K = F l w f P K (8)
V = F l w f P V (9)
其中 P Q 、 P K 、和 P V 是在不同窗口间共享的投影矩阵， Q , K , V ∈ R M 2 × d 。
LFEB作为LFCSTM的CNN分支部分，其主要作用是对输入图像的浅层特征 F s f 做进一步提取，逐步细化提取的特征，得到局部特征 F l f 。在卷积层部分，我们选择使用GeLU激活函数，相比于ReLU激活函数，GeLU激活函数引入了随机正则的思想，直观上更符合自然的认识。为了进一步提升特征的表达能力，考虑到模型的性能和复杂度，我们在卷积层之后引入了计算复杂度低且能保持高性能的高效通道注意力(ECA)机制，具体结构如图2(a)所示。同时，为了缓解梯度消失的问题，我们在卷积层之后加入了残差连接，LFEB的整体流程可以表示为
F l f 1 = f c o n v 3 ( G E L U ( f c o n v 3 ( F s f ) ) ) (13)
F l f 2 = f e c a ( F l f 1 ) (14)
F l f = F l f 1 × F l f 2 (15)
其中， f e c a ( ⋅ ) 为ECA机制。当得到局部特征 F l f 后，我们利用增强空间注意力(ESA)块将 F l f 和 F l w f 进行特征融合，从而得到更精确的特征，这一个步骤可以表示为
F n = f e s a ( F l f + F l w f ) (16)
其中 f e s a ( ⋅ ) 表示ESA机制，结构如图2(c)所示。至此，我们得到了LFCSTM的输出 F n 以作为后续模块的输入。
图3. 多路径动态卷积块(MDCB)
3.4. 多路径动态卷积块(MDCB)
针对参数量较少的轻量级网络，为了更好的学习图像的边缘信息，这需要对模型进行更精细的设计。Lu等人在ESRT中提出了使用高频滤波模块(High-Frequency Filtering Module, HFM) [19] 来保留特征图的细节和边缘，这不可避免的增加了模型的计算复杂度和特征冗余，从而导致视觉上不自然的SR图像。因此，我们基于动态卷积的思想，设计出了一个多路径动态卷积块(MDCB)作为整体模型的边缘增强分支，目的是有效的提取重要的边缘信息，同时过滤掉无用的特征。我们提出的MDCB以可接受的计算成本和参数量获得了比ESRT更好的性能和视觉效果。如图3所示，给定大小为 H × W × C 的浅层特征 F s f 作为输入
F t f = f c o n v 1 ( F s f ) (17)
其中 f c o n v 1 ( ⋅ ) 表示1 × 1卷积层，我们利用1 × 1卷积层作为过渡层，得到过渡特征 F t f ，这便于后续更有效的提取特征。然后，我们将过渡特征 F t f 作为后续三条路径的输入，其中path1和path3由单个的3 × 3卷积层组成，path2为卷积组，由级联的3 × 3卷积层、LeakyReLU函数、1 × 1卷积层和Sigmoid函数组成。值得注意的是，path2和path3将通过逐元素相乘的方式进行特征聚合，再通过一个3 × 3卷积层输入到动态卷积部分。至此，我们通过多路径进行特征提取的流程可以表示为
F t f 1 = f p a t h 1 ( F t f ) (18)
F t f 23 = f c o n v 3 ( f p a t h 2 ( F t f ) × ( f p a t h 3 ( F t f ) ) ) (19)
其中， F t f 1 和 F t f 23 分别表示经path1、path2和path3得到的特征信息， f p a t h 1 ( ⋅ ) 、 f p a t h 2 ( ⋅ ) 和 f p a t h 3 ( ⋅ ) 分别表示path1、path2和path3路径。在获得 F t f 1 和 F t f 23 后，我们采用动态卷积来进行边缘信息的学习。如图3中Attention部分所示，利用注意力机制得到接下来每个卷积核的权重。Attention部分与SENet [20] 类似，不同点在于最后采用SoftMax函数来生成两个自适应权重，动态的调节卷积核参数来提取图像的边缘特征。给定输入过渡特征 F t f ，动态卷积的具体操作如下
w 1 , w 2 = f s o f t ( F C ( R e L U ( F C ( f g a p ( F t f ) ) ) ) ) (20)
F e f = f c o n v 1 ( F t f 23 × w 1 + F t f 1 × w 2 ) + F s f (21)
其中， f g a p ( ⋅ ) 表示全局平均池化(Global Average Pooling, GAP)操作， F C ( ⋅ ) 表示全连接层， R e L U ( ⋅ ) 表示ReLU函数， f s o f t ( ⋅ ) 表示SoftMax函数。 w 1 和 w 2 分别表示两个自适应权重值。最后，我们通过一个残差连接得到边缘特征 F e f 用于SR图像重建。
Effects of components on model performance on Urban100 (×4
Scale
MDCB
ESA
LFEB
PSNR/SSIM
×4
×
26.25/0.7913
×
×
26.16/0.7890
×
×
×
26.15/0.7884
表2. 在Urban100 (×4)上各组件对模型性能影响
基金项目
四川省科技厅项目(编号：2021Z005)。
文章引用
李光明,张 倩,金 瑾,何 嘉. 基于CNN和Transformer的轻量级超分辨率重建网络研究Research on Lightweight Super-Resolution Network Based on CNN and Transformer[J]. 计算机科学与应用, 2023, 13(01): 93-103. https://doi.org/10.12677/CSA.2023.131010
参考文献ReferencesLedig, C., Theis, L., Huszar, F., et al. (2016) Photo-Realistic Single Image Super-Resolution Using a Generative Adver-sarial Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 105-114. <br>https://doi.org/10.1109/CVPR.2017.19Kim, J., Lee, J.K. and Lee, K.M. (2016) Deep-ly-Recursive Convolutional Network for Image Super-Resolution. 2016 IEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), Las Vegas, 27-30 June 2016, 1637-1645.
<br>https://doi.org/10.1109/CVPR.2016.181Ying, T., Jian, Y. and Liu, X. (2017) Image Super-Resolution via Deep Recursive Residual Network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Hon-olulu, 21-26 July 2017, 2790-2798.Ahn, N., Kang, B. and Sohn, K.A. (2018) Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network.Zheng, H., Wang, X. and Gao, X. (2018) Fast and Accurate Single Image Super-Resolution via Information Distillation Network.Hui, Z., Gao, X., Yang, Y., et al. (2019) Lightweight Image Super-Resolution with Information Multi-Distillation Network. Proceedings of the 27th ACM Inter-national Conference on Multimedia, Nice, 21-25 October 2019, 2024-2032. <br>https://doi.org/10.1145/3343031.3351084Liu, J., Tang, J. and Wu, G. (2020) Residual Feature Distillation Network for Lightweight Image Super-Resolution. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recog-nition (CVPR), Seattle, 13-19 June 2020, 2356-2365. <br>https://doi.org/10.1109/CVPR42600.2020.00243Vaswani, A., Shazeer, N., Parmar, N., et al. (2017) Atten-tion Is All You Need.Liu, Z., Lin, Y., Cao, Y., et al. (2021) Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, 10-17 Octo-ber 2021, 9992-10002.
<br>https://doi.org/10.1109/ICCV48922.2021.00986Liu, J., Zhang, W., Tang, Y., et al. (2020) Residual Feature Aggregation Network for Image Super-Resolution. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recog-nition (CVPR), Seattle, 13-19 June 2020, 2356-2365.
<br>https://doi.org/10.1109/CVPR42600.2020.00243Wang, Q., Wu, B., Zhu, P., et al. (2020) ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. 2020 IEEE/CVF Conference on Computer Vision and Pat-tern Recognition (CVPR), Seattle, 13-19 June 2020, 11531-11539. <br>https://doi.org/10.1109/CVPR42600.2020.01155Chen, Y., Dai, X., Liu, M., et al. (2020) Dynamic Convolu-tion: Attention Over Convolution Kernels. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 13-19 June 2020, 11027-11036.
<br>https://doi.org/10.1109/CVPR42600.2020.01104Dong, C., Loy, C.C., He, K., et al. (2016) Image Su-per-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, 295-307. <br>https://doi.org/10.1109/TPAMI.2015.2439281Lim, B., Son, S., Kim, H., et al. (2017) Enhanced Deep Residual Networks for Single Image Super-Resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, 21-26 July 2017, 1132-1140. <br>https://doi.org/10.1109/CVPRW.2017.151Zhang, Y., Li, K., Li, K., et al. (2018) Image Super-Resolution Using Very Deep Residual Channel Attention Networks. 15th European Conference, Munich, 8-14 September 2018, 294-310.
<br>https://doi.org/10.1007/978-3-030-01234-2_18Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020) An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.Chen, H., Wang, Y., Guo, T., et al. (2020) Pre-Trained Image Processing Transformer. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, 20-25 June 2021, 12294-12305.
<br>https://doi.org/10.1109/CVPR46437.2021.01212Liang, J., Cao, J., Sun, G., et al. (2021) SwinIR: Image Restoration Using Swin Transformer. 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, 11-17 October 2021, 1833-1844.
<br>https://doi.org/10.1109/ICCVW54120.2021.00210Lu, Z., Li, J., Liu, H., et al. (2022) Transformer for Single Image Super-Resolution. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Orleans, 19-20 June 2022, 456-465.
<br>https://doi.org/10.1109/CVPRW56347.2022.00061Jie, H., Li, S. and Gang, S. (2018) Squeeze-and-Excitation Networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 18-23 June 2018, 7132-7141.Cai, J., Gu, S., Timofte, R., et al. (2019) NTIRE 2019 Challenge on Real Image Su-per-Resolution: Methods and Results. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Work-shops (CVPRW), Long Beach, 16-17 June 2019, 2211-2223.Bevilacqua, M., Roumy, A., Guillemot, C., et al. (2012) Low-Complexity Single-Image Super-Resolution Based on Nonnegative Neighbor Embedding. Proceedings British Machine Vision Conference 2012, Surrey, 3-7 September 2012, 135.1-135.10. <br>https://doi.org/10.5244/C.26.135Zeyde, R., Elad, M. and Protter, M. (2010) On Single Image Scale-Up Using Sparse-Representations. Curves and Surfaces—7th International Conference, Avignon, 24-30 June 2010, 711-730.Martin, D., Fowlkes, C., Tal, D., et al. (2002) A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. IEEE International Conference on Computer Vision, Vancouver, 7-14 July 2001, 416-423.Huang, J.B., Singh, A. and Ahuja, N. (2015) Single Image Super-Resolution from Transformed Self-Exemplars. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, 7-12 June 2015, 5197-5206.
<br>https://doi.org/10.1109/CVPR.2015.7299156Aizawa, K., Fujimoto, A., Otsubo, A., et al. (2020) Building a Manga Dataset “Manga109” with Annotations for Multimedia Applications. IEEE MultiMedia, 27, 8-18. <br>https://doi.org/10.1109/MMUL.2020.2987895Kim, J., Lee, J.K. and Lee, K.M. (2016) Accurate Image Su-per-Resolution Using Very Deep Convolutional Networks. IEEE Conference on Computer Vision & Pattern Recognition, Las Vegas, 27-30 June 2016, 1646-1654.
<br>https://doi.org/10.1109/CVPR.2016.182Lai, W.S., Huang, J.B., Ahuja, N., et al. (2017) Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 5835-5843.
<br>https://doi.org/10.1109/CVPR.2017.618
