Light field gaussian splatting representation and joint spatial-angular super-resolution for low-altitude scenarios
-
摘要:
光场成像技术可同时捕获光线的空间位置和角度信息,在无人机感知、环境监测等低空经济应用场景中具有广阔应用前景。然而,现有光场相机成像及超分辨率方法难以兼顾空间分辨率与角度分辨率,制约了其在精确感知和高精度建模等任务中的应用。为此,提出一种面向低空场景的光场高斯泼溅表示及光场空间-角度同时超分辨率(LFASSR)方法。将传统3D高斯泼溅建模方法拓展到4D光场表示空间中,提出一种4D光场高斯泼溅表示方法,并利用光场极平面(EPI)的空间-角度几何特性,将其分解为水平与垂直方向上的2D高斯泼溅级联建模,以更有效地表征光场几何特征。基于此,构建了一个通用的光场图像超分辨率框架,在提取并融合光场空间、角度及EPI域特征后,构建水平和垂直方向的光场几何特征空间,并分别进行2D高斯泼溅建模,最终经EPI渲染,生成致密视角且高空间分辨率的高质量光场图像。在公开光场图像数据集及低空场景光场数据上的实验结果表明:所提方法在LFASSR任务中显著优于现有方法,在峰值信噪比(PSNR)和结构相似性指标(SSIM)等客观指标上均达到当前最优水平,在视觉质量上展现出清晰的细节恢复能力与自然的视差过渡效果。在独立的光场空间超分辨率(LFSSR)任务和光场角度超分辨率(LFASR)任务中,所提方法同样展现出良好的泛化能力与竞争力。
-
关键词:
- 光场 /
- 光场超分辨率 /
- 光场高斯泼溅 /
- 空间-角度同时超分辨率 /
- 极平面几何
Abstract:Unmanned aerial vehicle (UAV) perception and environmental monitoring are two low-altitude commercial applications where light field imaging technology shows significant promise by simultaneously capturing the spatial and angular information of light rays. However, existing light field cameras and super-resolution methods struggle to jointly optimize spatial and angular resolutions, limiting their effectiveness in tasks requiring precise perception and high-fidelity reconstruction. To address this challenge, this paper proposes a light field Gaussian splatting representation and a light field angular spatial super-resolution (LFASSR) method tailored for low-altitude scenarios. Specifically, we extend the conventional 3D Gaussian splatting framework into the 4D light field domain and propose a 4D light field Gaussian splatting representation. Leveraging the geometric structure of light field epipolar plane images (EPIs), we decompose the 4D representation into cascaded 2D Gaussian splatting models along horizontal and vertical directions, enabling more efficient modeling of light field geometry. Based on this representation, we develop a unified light field super-resolution framework. The framework first extracts and fuses features from the spatial, angular, and EPI domains, then constructs directional geometric feature spaces for horizontal and vertical views, where 2D Gaussian splatting is applied separately. Finally, high-quality dense-view light field images with enhanced spatial resolution are rendered via EPI-based synthesis. The suggested methodology greatly outperforms state-of-the-art methods in joint spatial-angular super-resolution, as shown by extensive experimentation on public light field benchmarks and a recently gathered low-altitude light field dataset. It achieves the best performance in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), while also producing visually superior results with sharp detail recovery and smooth, natural disparity transitions. Moreover, the method exhibits strong generalization capability and competitive performance in standalone light field spatial super-resolution (LFSSR) and light field angular super-resolution (LFASR) tasks.
-
表 1 本文模型与7种模型在空间(×2)-角度(2×2→5×5)同时超分辨率任务上的性能对比
Table 1. Performance comparisons of this paper with seven models on the joint spatial(×2) and angular(2×2→5×5) super-resolution task
LFASR模型 LFSSR模型 PSNR/dB,SSIM 30Scenes Occlusion Reflective HCInew HCIold EPFL DistgASR[16] DistgSSR[16] 41.93 / 0.992 0 38.36 / 0.985 4 38.94 / 0.977 7 34.74 / 0.961 7 41.56 / 0.990 5 32.90 / 0.969 5 DistgASR[16] EPIT[25] 41.85 / 0.991 8 38.27 / 0.985 1 38.89 / 0.976 8 35.21 / 0.965 3 42.12 / 0.991 4 33.21 / 0.970 3 DistgASR[16] MLFSR[14] 41.99 / 0.992 1 38.41 / 0.985 7 39.01 / 0.977 2 34.87 / 0.962 8 41.98 / 0.991 2 33.37 / 0.970 8 LF-EASR[7] DistgSSR[16] 41.86 / 0.991 9 38.21 / 0.985 1 38.94 / 0.977 6 34.66 / 0.961 2 41.67 / 0.990 4 33.17 / 0.969 7 LF-EASR[7] EPIT[25] 41.80 / 0.991 7 38.15 / 0.984 9 38.93 / 0.977 4 35.06 / 0.964 4 42.05 / 0.991 1 33.32 / 0.969 5 LF-EASR[7] MLFSR[14] 41.77 / 0.991 7 38.12 / 0.984 9 38.91 / 0.977 2 34.67 / 0.961 3 41.68 / 0.990 7 33.28 / 0.970 0 LFSR[19] LFSR[19] 41.89 / 0.991 9 38.30 / 0.985 3 38.86 / 0.977 2 34.46 / 0.959 9 41.72 / 0.990 7 32.98 / 0.969 2 本文模型 本文模型 42.15 / 0.992 3 38.64 / 0.986 2 39.22 / 0.978 2 35.35 / 0.965 3 42.81 / 0.992 5 33.67 / 0.971 9 注:指标最优和次优分别作加粗和下划线处理。 表 2 本文与7种模型在LFASR(2×2→7×7)任务上的PSNR和SSIM对比
Table 2. Performance comparisons on PSNR and SSIM of this paper with seven models in the LFASR task(2×2→7×7)
模型 发表年份 PSNR/dB,SSIM HCInew HCIold 30Scenes Occlusion Reflective LF-EASR[7] 2022 35.86 / 0.975 41.54 / 0.971 43.44 / 0.995 39.80 / 0.992 39.35 / 0.981 DistgASR[16] 2022 34.70 / 0.974 42.18 / 0.978 43.67 / 0.995 39.46 / 0.991 39.11 / 0.978 HLFSR-ASR[17] 2023 36.43 / 0.982 43.25 / 0.990 44.03 / 0.996 39.87 / 0.992 39.60 / 0.981 Liu等[26] 2023 35.29 / 0.954 41.92 / 0.969 43.33 / 0.988 39.67 / 0.983 39.20 / 0.963 Li和Zhong[27] 2024 35.21 / 0.975 42.36 / 0.979 43.95 / 0.996 39.68 / 0.991 39.36 / 0.981 Liu等[28] 2024 35.41 / 0.954 42.02 / 0.969 43.38 / 0.995 39.71 / 0.991 39.31 / 0.982 ViewFormer[29] 2025 35.37 / 0.977 41.93 / 0.975 43.82 / 0.995 39.86 / 0.992 39.57 / 0.981 本文 37.71 / 0.983 43.99 / 0.993 44.15 / 0.995 40.07 / 0.992 39.45 / 0.984 注:指标最优和次优分别作加粗和下划线处理。 表 3 本文与9种模型在LFSSR任务(放大2倍和4倍)上的PSNR和SSIM对比
Table 3. Performance comparisons on PSNR and SSIM of this paper with nine models in the LFSSR tasks(×2 and ×4)
模型 发表年份 PSNR/dB,SSIM EPFL HCInew HCIold INRIA STFgantry 放大2倍 放大4倍 放大2倍 放大4倍 放大2倍 放大4倍 放大2倍 放大4倍 放大2倍 放大4倍 DPT[30] 2022 34.49 28.94 37.36 31.20 44.30 37.41 36.41 30.96 39.43 31.15 0.975 8 0.917 0 0.977 1 0.918 8 0.994 3 0.972 1 0.984 3 0.952 3 0.992 6 0.948 8 DistgSSR[16] 2022 34.81 28.99 37.96 31.38 44.94 37.56 36.59 30.99 40.40 31.65 0.978 7 0.919 5 0.979 6 0.921 7 0.994 9 0.973 2 0.985 9 0.951 9 0.994 2 0.953 5 EPIT[25] 2023 34.83 29.34 38.23 31.51 45.08 37.68 36.67 31.37 42.17 32.18 0.977 5 0.919 7 0.981 0 0.923 1 0.994 9 0.973 7 0.985 3 0.952 6 0.995 7 0.957 1 HLFSR-SSR[17] 2023 35.31 29.20 38.32 31.57 44.98 37.78 37.06 31.24 40.85 31.64 0.980 0 0.922 2 0.980 7 0.923 8 0.995 0 0.972 4 0.986 7 0.953 4 0.994 7 0.953 7 LF-DET[31] 2023 35.26 29.47 38.31 31.56 44.99 37.84 36.95 31.39 41.76 32.14 0.979 7 0.923 0 0.980 7 0.923 5 0.995 0 0.974 4 0.986 4 0.953 4 0.995 5 0.957 3 徐欣宜等[32] 2023 34.49 28.70 37.47 31.13 44.25 37.21 36.37 30.73 38.89 30.61 0.976 7 0.914 8 0.979 0 0.920 8 0.994 3 0.972 4 0.984 3 0.950 3 0.991 6 0.943 2 MLFSR[14] 2024 35.22 29.28 38.14 31.56 44.90 37.83 36.92 31.24 40.98 32.03 0.980 1 0.921 8 0.980 3 0.923 5 0.995 0 0.974 5 0.986 5 0.953 1 0.994 9 0.956 7 LF-TGUnet[33] 2024 34.67 28.96 37.80 31.42 44.78 37.63 36.46 31.28 39.95 31.36 0.977 6 0.918 9 0.979 1 0.921 5 0.994 8 0.973 4 0.985 7 0.952 9 0.993 6 0.951 1 LF-GIANet[34] 2025 34.56 29.14 38.01 31.40 44.80 37.60 36.45 31.25 40.45 31.30 0.978 2 0.919 8 0.979 8 0.921 8 0.994 8 0.973 3 0.985 8 0.952 2 0.994 5 0.950 5 本文 35.70 29.47 38.49 31.61 45.31 37.86 37.28 / 31.45 41.45 31.72 0.980 9 0.924 4 0.981 5 0.924 4 0.995 3 0.974 8 0.987 2 0.955 0 0.995 3 0.955 3 注:指标最优和次优分别作加粗和下划线处理。 表 4 低空场景光场图像的空间(×2)-角度(2×2→5×5)同时超分辨率性能对比
Table 4. Performance comparisons on the joint spatial (×2) and angular (2×2→5×5) super-resolution for low-altitude light field images
场景编号 PSNR/dB,SSIM DistgASR[16]
+DistgSSR[16]DistgASR[16]
+EPIT[25]DistgASR[16]
+MLFSR[14]LF-EASR[7]
+DistgSSR[16]LF-EASR[7]
+EPIT[25]LF-EASR[7]
+ MLFSR[14]LFSR[19] 本文模型 Image36 37.09/0.981 7 37.06/0.981 6 37.15/0.981 9 36.85/0.980 4 37.01/0.981 3 37.00/0.980 9 36.82/0.980 4 37.20/0.982 3 Image62 36.34/0.978 5 36.41/0.979 1 36.64/0.979 3 36.07/0.977 7 36.42/0.978 8 36.34/0.978 2 36.04/0.977 5 36.64/0.979 3 Image63 36.08/0.983 4 36.25/0.984 0 36.30/0.984 1 36.17/0.983 9 36.19/0.983 8 36.13/0.983 6 35.89/0.982 8 36.44/0.984 5 Image81 37.87/0.980 7 37.97/0.981 2 37.92/0.980 9 37.95/0.981 2 37.87/0.980 8 37.85/0.980 6 37.68/0.980 0 38.04/0.981 4 Image82 36.87/0.978 7 37.01/0.979 3 37.03/0.979 2 36.68/0.977 7 36.93/0.978 4 36.71/0.977 8 36.50/0.977 1 37.30/0.980 5 Image85 37.56/0.986 1 37.79/ 0.9865 37.74/ 0.9863 37.41/ 0.9857 37.76/ 0.9860 37.42/ 0.9855 37.10/ 0.9847 38.10/ 0.9872 Image86 39.56/0.978 9 39.57/ 0.9789 39.48/ 0.9786 39.59/ 0.9790 39.52/ 0.9787 39.55/ 0.9788 39.46/ 0.9785 39.64/ 0.9792 Image89 37.26/0.978 9 37.23/ 0.9790 37.33/ 0.9792 37.20/ 0.9787 37.26/ 0.9789 37.23/ 0.9787 37.03/ 0.9782 37.37/ 0.9795 Image98 37.21/0.983 7 37.41/ 0.9844 37.29/ 0.9839 37.06/ 0.9831 37.29/ 0.9838 37.11/ 0.9831 37.07/ 0.9832 37.36/ 0.9843 Image128 39.52/0.991 0 39.69/ 0.9920 39.56/ 0.9921 39.51/ 0.9919 39.69/ 0.9918 39.53/ 0.9916 39.14/ 0.9901 39.99/ 0.9928 Image187 37.23/ 0.9914 37.38/ 0.9924 37.21/ 0.9920 37.48/ 0.9921 37.25/ 0.9916 37.10/ 0.9914 36.80/ 0.9910 37.69/ 0.9926 Image264 35.68/ 0.9812 36.04/ 0.9826 36.19/ 0.9830 34.69/ 0.9775 36.02/ 0.9819 34.62/ 0.9763 35.34/ 0.9798 36.01/ 0.9829 注:指标最优和次优分别作加粗和下划线处理;从左到右8个模型的平均值分别为37.36/ 0.9829 、37.48/0.9834 、37.49/0.9834 、37.22/0.9824 、37.43/0.9830 、37.22/0.9822 、37.07/0.9820 、37.65/0.9839 。表 5 LFASR和LFSSR消融实验
Table 5. Ablation study on the LFASR and LFSSR tasks
超分辨率任务类型 数据集 PSNR (dB)/SSIM w/o 4DLFGS 本文方法 LFASR HCInew 34.27/0.970 0 37.71/0.982 9 HCIold 42.27/0.979 0 43.99/0.993 1 LFSSR HCInew 38.32/0.981 0 38.49/0.981 5 HCIold 45.01/ 0.9950 45.31/ 0.9953 表 6 LFASSR任务消融实验
Table 6. Ablation study on the LFASSR task
交叉注意力
模块自注意力
模块4DLFGS PSNR/dB,SSIM 参数量 浮点运算速度/
(109·s−1)HCInew HCIold EPFL 30Scenes Occlusions Reflective √ 34.55/0.960 4 41.88/0.991 2 33.22/0.970 2 41.84/0.991 8 38.28/0.985 3 38.96/0.977 3 1.483×107 76.12 √ 34.41/0.959 6 41.70/0.991 2 33.26/0.969 9 41.90/0.991 9 38.34/0.985 5 39.06/0.978 1 1.485×107 76.30 √ √ 35.12/0.963 9 42.40/0.992 0 33.51/0.971 1 41.96/0.992 0 38.45/0.985 7 39.12/0.978 6 1.638×107 85.63 √ √ 35.09/ 0.9639 42.42/ 0.9920 33.51/ 0.9715 42.11/ 0.9922 38.53/ 0.9858 39.13/ 0.9780 1.525×107 119.93 √ √ √ 35.35/ 0.9653 42.81/ 0.9925 33.67/ 0.9719 42.15/ 0.9923 38.64/ 0.9862 39.22/ 0.9782 1.678×107 129.26 -
[1] Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[D]. Palo Alto: Stanford University, 2005: 5-11. [2] Lyu W Q, Sheng H, Ke W, et al. Advances in light field spatial super-resolution: a comprehensive literature survey[J]. IEEE Access, 2025, 13: 18470-18497. [3] Hu H, Guo M T, Li H D, et al. Revisiting spatio-angular trade-off in light field cameras and extended applications in super-resolution[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(6): 3019-3033. [4] Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras[J]. ACM Transactions on Graphics, 2016, 35(6): 1-10. [5] Zhang S, Chang S, Shen Z Q, et al. Micro-lens image stack upsampling for densely-sampled light field reconstruction[J]. IEEE Transactions on Computational Imaging, 2021, 7: 799-811. [6] Jin J, Hou J H, Yuan H, et al. Learning light field angular super-resolution via a geometry-aware network[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11141-11148. [7] Liu G S, Yue H J, Wu J M, et al. Efficient light field angular super-resolution with sub-aperture feature learning and macro-pixel upsampling[J]. IEEE Transactions on Multimedia, 2023, 25: 6588-6600. [8] Wang L Y, Ren L F, Wei X Y, et al. Light field angular super-resolution based on intrinsic and geometric information[J]. Knowledge-Based Systems, 2023, 270: 110553. [9] Mao Y F, Xiao Z Y, An P, et al. Deep sparse-to-dense inbetweening for multi-view light fields[J]. IEEE Transactions on Image Processing, 2025, 34: 6302-6317. [10] Yuan Y, Cao Z Q, Su L J. Light-field image superresolution using a combined deep CNN based on EPI[J]. IEEE Signal Processing Letters, 2018, 25(9): 1359-1363. [11] Wang Y L, Liu F, Zhang K B, et al. LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution[J]. IEEE Transactions on Image Processing, 2018, 27(9): 4274-4286. [12] Zhang S, Lin Y F, Sheng H. Residual networks for light field image super-resolution[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2020: 11038-11047. [13] Liang Z Y, Wang Y Q, Wang L G, et al. Light field image super-resolution with transformers[J]. IEEE Signal Processing Letters, 2022, 29: 563-567. [14] Gao R S, Xiao Z Y, Xiong Z W. Mamba-based light field super-resolution withEfficient subspace scanning[C]//Proceedings of the Computer Vision–ACCV 2024. Berlin: Springer, 2024: 421-437. [15] Chen S R, Chen L, Wu D F, et al. Enhancing light field image super-resolution through Mamba-based spatial–angular correlation learning[J]. The Visual Computer, 2025, 41(15): 12649-12662. [16] Wang Y Q, Wang L G, Wu G C, et al. Disentangling light fields for super-resolution and disparity estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(1): 425-443. [17] Van Duong V, Huu T N, Yim J, et al. Light field image super-resolution network via joint spatial-angular and epipolar information[J]. IEEE Transactions on Computational Imaging, 2023, 9: 350-366. [18] Yoon Y, Jeon H G, Yoo D, et al. Learning a deep convolutional network for light-field image super-resolution[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop. Piscataway: IEEE Press, 2016: 57-65. [19] Liu G S, Yue H J, Li K, et al. Adaptive pixel aggregation for joint spatial and angular super-resolution of light field images[J]. Information Fusion, 2024, 104: 102183. [20] Liu D Y, Li S Z, Mao Y F, et al. Learning implicit and detail-enhanced network for light field image spatial-angular super-resolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2026, 36(2): 1544-1557. [21] Wu T, Yuan Y J, Zhang L X, et al. Recent advances in 3D Gaussian splatting[J]. Computational Visual Media, 2024, 10(4): 613-642. [22] Hu J T, Xia B, Chen B, et al. GaussianSR: high fidelity 2D Gaussian splatting for arbitrary-scale image super-resolution[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(4): 3554-3562. [23] Zhang H, Zhou W H, Lin L L, et al. Cascade residual learning based adaptive feature aggregation for light field super-resolution[J]. Pattern Recognition, 2025, 165: 111616. [24] Sheng H, Cong R X, Yang D, et al. UrbanLF: a comprehensive light field dataset for semantic segmentation of urban scenes[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(11): 7880-7893. [25] Liang Z Y, Wang Y Q, Wang L G, et al. Learning non-local spatial-angular correlation for light field image super-resolution[C]// Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2023: 12342-12352. [26] Liu D Y, Mao Y F, Zhou X F, et al. Learning a multilevel cooperative view reconstruction network for light field angular super-resolution[C]// Proceedings of the 2023 IEEE International Conference on Multimedia and Expo. Piscataway: IEEE Press, 2023: 1271-1276. [27] Li D C, Zhong R. Light field image angular super-resolution using edge features[C]//Proceedings of the 2024 6th International Conference on Robotics and Computer Vision. Piscataway: IEEE Press, 2024: 132-138. [28] Liu D Y, Mao Y F, Zuo Y F, et al. Light field angular super-resolution network based on convolutional transformer and deep deblurring[J]. IEEE Transactions on Computational Imaging, 2024, 10: 1736-1748. [29] Wang S Z, Lu Y, Xia W, et al. Light field angular super-resolution by view-specific queries[J]. The Visual Computer, 2025, 41(5): 3565-3580. [30] Wang S Z, Zhou T F, Lu Y, et al. Detail-preserving transformer for light field image super-resolution[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(3): 2522-2530. [31] Cong R X, Sheng H, Yang D, et al. Exploiting spatial and angular correlations with deep efficient transformers for light field image super-resolution[J]. IEEE Transactions on Multimedia, 2023, 26: 1421-1435. [32] 徐欣宜, 邓慧萍, 向森, 等. 基于特征交互融合与注意力的光场图像超分辨率[J]. 激光与光电子学进展, 2023, 60(14): 1410017.Xu X Y, Deng H P, Xiang S, et al. Light field image super-resolution based on feature interaction fusion and attention mechanism[J]. Laser & Optoelectronics Progress, 2023, 60(14): 1410017(in Chinese). [33] 黄莉, 吕天琪, 武迎春, 等. 基于双路引导更新的光场图像超分网络[J]. 光电工程, 2024, 51(12): 240222.Huang L, Lv T Q, Wu Y C, et al. Two-way guided updating network for light field image super-resolution[J]. Opto-Electronic Engineering, 2024, 51(12): 240222(in Chinese). [34] Zhang W, Li H J, Ke W. LF-GIANet: cascaded global-view information adaptation-guided network for light field image super-resolution[J]. Multimedia Systems, 2025, 31(2): 73. [35] Chen Y B, Liu S F, Wang X L. Learning continuous image representation with local implicit image function[C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2021: 8624-8634. -


下载: