Neural network approach to ensuring visual coherence in augmented reality flight simulators
https://doi.org/10.26467/2079-0619-2024-27-4-8-19
Abstract
In 2023, the leading US aerospace corporation Lockheed Martin announced the simultaneous development of several extended/augmented reality (XR/AR) simulators for pilots of TF-50, F-16, F-22, and F-35 without being a pioneer in this area of focus, in 2022 similar projects were launched by Boeing and the leading British aeronautical equipment manufacturer BAE Systems. In January 2024 the US Air Force invested in the development of pilot AR simulators based on Microsoft Hololens augmented reality smart glasses. At the same time, Apple began bulk sales of the Apple Vision Pro AR headset. It is difficult to doubt that in 2024 a variety of new aviation simulators will appear using this device. The rapid development of a new generation of aerospace simulator technology, i.e., XR/AR simulators, is accompanied by a boom in research in the field of visual coherence (VC) of augmented reality scenes: virtual objects in these scenes should be virtually identical with real ones. It is VC that provides new capabilities of AR simulators, which fundamentally distinguish from conventional flight simulators with virtual reality. Recently, VC has been increasingly provided by neural network methods, thereby, the most important aspects of VC are lighting conditions, so the major share of research is focused on transferring these conditions (location of light sources and their color tone) from the real world to the virtual one, but the great body of the known approaches are characterized by the lack of versatility and the need to perform manual procedures. These disadvantages are not found in the spectral transplantation method based on twodimensional spectral image conversions, which, however, requires determining the size of the spectrum part being transplanted from the real picture of the world to a virtual object. This article is devoted to the development of a neural network model for the mechanism of selecting the optimal size of a spectral transplant.
About the Authors
A. L. Gorbunov
Moscow State Technical University of Civil Aviation
Russian Federation
Russian Federation
Andrey L. Gorbunov, Candidate of Technical Sciences, Associate Professor, Associate Professor of the Air Traffic Management Chair
Moscow
Yunhan Li
Moscow State Technical University of Civil Aviation
Russian Federation
Russian Federation
Yunhan Li, Postgraduate Student
Moscow
References
1. Gorbunov, A.L. (2016). Trainer of airport the specials-technician. Civil Aviation High Technologies, no. 225 (3), pp. 92–97. (in Russian)
2. Gorbunov, А.L. (2023). Visual coherence for augmented reality. Advanced Engineering Research, no. 2, pp. 180–190. DOI: 10.23947/2687-1653-2023-23-2-180-190 (in Russian)
3. Tewari, A., Fried, O., Thies, J. et al. (2020). State of the art on neural rendering. Computer Graphics Forum, vol. 39, issue 2, pp. 701–727. DOI: 10.1111/cgf.14022
4. Einabadi, F., Guillemaut, J., Hilton, A. (2021). Deep neural models for illumination estimation and relighting: A survey. Computer Graphics Forum, vol. 40, issue 6, pp. 315–331. DOI: 10.1111/cgf.14283
5. Ghasemi, Y., Jeong, H., Choi, S., Lee, J., Park, K. (2022). Deep learning-based object detection in augmented reality: A systematic review. Computers in Industry, vol. 139, ID: 103661. DOI: 10.1016/j.compind.2022.103661 (accessed: 05.01.2024).
6. Tu, W.-C., He, Q., Chien, S.-Y. (2016). Real-time salient object detection with a minimum spanning tree. In: Proceedings of IEEE CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 2334–2342.
7. Yang, J., Yang, M.-H. (2012). Topdown visual saliency via joint CRF and dictionary learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. USA, RI, pp. 2296–2303. DOI: 10.1109/CVPR.2012.6247940 (accessed: 05.01.2024).
8. Rosin, P.L. (2009). A simple method for detecting salient regions. Pattern Recognition, vol. 42, issue 11, pp. 2363–2371. DOI: 10.1016/j.patcog.2009.04.021
9. Girshick, R., Donahue, J., Darrel, T., Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, USA, OH, pp. 580–587. DOI: 10.1109/CVPR.2014.81
10. Krähenbühl, P., Koltun, V. (2014). Geodesic object proposals. In: Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science. Springer, Cham, vol. 8693. DOI: 10.1007/978-3-319-10602-1_47 (accessed: 05.01.2024).
11. Pont-Tuset, J., Arbeláez, P., Barron, J.T., Marques, F., Malik, J. (2017). Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 1, pp. 128–140. DOI: 10.1109/TPAMI.2016.2537320 (accessed: 05.01.2024).
12. Zitnick, C.L., Dollar, P. (2014). Edge boxes: Locating object proposals from edges. In: Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science. Springer, Cham, vol. 8693. DOI: 10.1007/978-3-319-10602-1_26 (accessed: 05.01.2024).
13. Kuo, W., Hariharan, B., Malik, J. (2015). Deepbox: Learning objectness with convolutional networks. In: IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 2479–2487. DOI: 10.1109/ICCV.2015.285 (accessed: 05.01.2024).
14. Pinheiro, P.O., Lin, T.-Y., Collobert, R., Dollar, P. (2016). Learning to refine object segment. In: Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, Springer, Cham, vol. 9905. DOI: 10.1007/978-3-319-46448-0_5 (accessed: 05.01.2024).
15. Redmon, J., Divvala, S., Girshick, R., Farhadi, A. (2016). You only look once: Unified, real-time object detection. In: Proceedings of IEEE CVF Computer Vision and Pattern Recognition Conference (CVPR), USA, Las Vegas, NV, pp. 779–788. DOI: 10.1109/CVPR.2016.91 (accessed: 05.01.2024).
16. Shanmugamani, R. (2018). Deep learning for computer vision. Packt Publishing, 310 p.
17. Zhao, Z.-Q., Zheng, P., Xu, S.-T., Wu, X. (2019). Object detection with deep learning: A review. IEEE Transactions on Neural Networks and Learning Systems, vol. 30, issue 11, pp. 3212–3232. DOI: 10.1109/TNNLS. 2018.2876865 (accessed: 05.01.2024).
18. Gatys, L., Ecker, A., Bethge, M. (2016). A Neural algorithm of artistic style. Journal of Vision, vol. 16, ID: 326. DOI: 10.1167/16.12.326 (accessed: 05.01.2024).
19. Changela, A., Zaveri, M., Verma, D. (2023). A Comparative study on CORDIC algorithms and applications. Journal of Circuits, Systems and Computers, vol. 32, no. 05, ID: 2330002. DOI: 10.1142/S0218126623300027 (accessed: 05.01.2024).
Review
For citations:
Gorbunov A.L., Li Yu. Neural network approach to ensuring visual coherence in augmented reality flight simulators. Civil Aviation High Technologies. 2024;27(4):8-19. (In Russ.) https://doi.org/10.26467/2079-0619-2024-27-4-8-19
Views:
473
Email this article 
