Bihan Wen, Saiprasad Ravishankar, Zhizhen Zhao, Raja Giryes, Jong Chul Ye
Thanks to the tremendous interest from the research community, the focus of the March issue of the IEEE Signal Processing Magazine is on the second volume of the special issue on physics-driven machine learning for computational imaging, which brings together nine articles of the 19 accepted papers from the original 47 submissions.
What makes computational imaging more exciting is its close relationship with real-world applications using imaging hardware. Unlike many other machine learning approaches to computer vision and image-processing problems that mainly deal with digitized images, machine learning approaches to computational imaging have their origins in real-world imaging hardware. Therefore, the physics-driven principle is tightly coupled to specific applications; hence, it is important to understand how the machine learning approaches can be integrated into the computational imaging pipeline using examples from real-world applications.
Therefore, the review and tutorial articles in this March special issue aim to provide an overview of the real-world applications of recently proposed physics-driven learning methods. Specifically, this edition aims to provide readers with more detailed information on how physics-based machine learning can be used to solve real-world imaging problems caused by electromagnetic (EM) waves, optics, and magnetic resonance imaging (MRI).
The first two articles focus on imaging problems from EM waves, which are widely applied in sensing for security, biomedicine, geophysics, and various industries. Specifically, the article by Guo et al. [A1] provides informative background on EM imaging and basic formulations of the inverse problem. The authors then focus on three types of strategies combining physics and machine learning for linear and nonlinear imaging and discuss their advantages and limitations. The next article, by Su et al. [A2], focuses more on terahertz frequency-band imaging problems. As terahertz waves can partially penetrate through varieties of optically opaque objects and the rotational, vibrational, torsional frequencies of a great variety of molecules fall in the terahertz regime, terahertz imaging has been extensively studied for industrial inspection, security screening, chemical inspection, and nondestructive evaluation. In this article, the authors provide a detailed survey how learning-based approaches can be utilized to terahertz image restoration and reconstruction.
As one might easily expect, a large part of the imaging problems in the real world arise from optics. The next three articles focus on the physics-driven machine learning approaches to solve optical imaging problems, such as optical phase retrieval, hyperspectral unmixing, and even optical element design problems. Specifically, Pinilla et al. [A3] discuss hybrid approaches of model-driven networks or deep unfolding for the recovery of a complex signal from phaseless data acquired in the form of its diffraction patterns. Another important application of optical imaging is in remote sensing, which typically collect data using hyperspectral imaging sensors, allowing the identification of materials based on their unique spectral signatures that go beyond their visible properties. Although the classical approaches to hyperspectral imaging rely on explicit mixing models, these methods may not be accurate due to their limited modeling capabilities, especially when analyzing real scenes with unknown complex physical properties. In the article by Chen et al. [A4], the authors focus on combining the advantages of both physics-based models and data-driven methods to address the challenges in the hyperspectral unmixing problem. The next article, by Arguello et al. [A5], gives an overview of a different but very exciting way to use physics-driven machine learning for optical imaging. In contrast to the other two articles on optical imaging, which mainly focus on image reconstruction for specific optics, the authors examine a current research trend in the optics community that uses machine learning approaches for the design of optical encoding elements to achieve improved imaging physics.
The remaining four articles mainly focus on real-world medical imaging systems, such as X-ray and MRI. In fact, X-ray imaging could be regarded as another type of EM or optical imaging problem with X-ray ranges. However, the fundamental difference arises from tomography, where multiple projection images from different angles can be combined to reconstruct the internal structure of the object through inverse Radon transform. However, the basic assumption of the inverse Radon transform is that there are enough high-quality projection images. Due to the potential for damaging healthy tissue, minimizing the radiation dose for X-ray computerized tomography has been extensively studied over the past two decades. The article by Xia et al. [A6] provides a survey of the physics-driven machine learning approaches to address the issue of high-quality tomographic reconstruction from low-quality projection data.
Although MRI also relies on the microwave range, compared to all of the imaging problems mentioned above, MRI is unique in that it is based on Fourier coding, which allows for much higher resolution beyond the diffraction limit. However, due to the nature of Fourier coding, the acquisition time of MRI is significantly slower compared to other imaging approaches, and the main investigations into computational imaging in MR focus on accelerated acquisition. The article by Lam et al. [A7] deals with even more challenging problems of MRI, known as the MR spectroscopic imaging. The challenge here lies in the additional spectroscopic dimension in addition to spatial coding for MRI. Therefore, a naive way to use Fourier coding requires a significant amount of acquisition time, making it difficult to use in real medical applications. Therefore, the authors survey recent advance in this field by utilizing the physics-driven machine learning approaches. The article by Zhu et al. [A8] discusses physics-driven machine learning approaches for another time-consuming MR acquisition model, quantitative MRI, which aims to obtain quantitative biophysical parameters based on physical models derived from MR spin magnetization evolution. Although deep learning has already become a key element of MR acceleration, one of its bottlenecks is overfitting to insufficient training data. The article by Yang et al. [A9] reviews an emerging paradigm, imaging physics-based data synthesis, that can provide huge training data in biomedical MR without or with few real data.
Although we have tried our best to provide a comprehensive overview of physics-driven machine learning approaches to computational imaging, we are aware that even the two volumes of the special issue are insufficient to cover many exciting developments in the field. While not sufficient, we hope that the overview of the theoretical foundations and practical applications can give readers a general overview of the field and encourage them to delve into this exciting area of research.
We would like to thank the anonymous reviewers and the editor for their careful reading and very important comments to make this special issue successful.
Bihan Wen (bihan.wen@ntu.edu.sg) received his B.Eng degree in electrical and electronic engineering from Nanyang Technological University, Singapore, in 2012 and his M.S and Ph.D. degrees in electrical and computer engineering from the University of Illinois at Urbana-Champaign in 2015 and 2018, respectively. He is a Nanyang assistant professor with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798. He serves as an associate editor of IEEE Transactions on Circuits and Systems for Video Technology. He was a recipient of the 2016 Yee Fellowship and the 2012 Professional Engineers Board Gold Medal of Singapore. He coauthored a paper that received the Best Paper Runner-Up Award at the IEEE International Conference on Multimedia and Expo in 2020. His research interests include machine learning, computer vision, image and video processing, and computational imaging. He is a Member of IEEE.
Saiprasad Ravishankar (ravisha3@msu.edu) received his B.Tech. degree in electrical engineering from the Indian Institute of Technology Madras, India, and his M.S. and Ph.D. degrees in electrical and computer engineering from the University of Illinois at Urbana-Champaign. He is an assistant professor in the Departments of Computational Mathematics, Science and Engineering, from 2015 to 2018, and Biomedical Engineering at Michigan State University, East Lansing, MI 48824 USA. He did postdoctoral research in the Department of Electrical Engineering and Computer Science at the University of Michigan from 2015 to 2018, and in the Theoretical Division at Los Alamos National Laboratory during 2018–2019. He organized special sessions or workshops on computational imaging at the 2016 IEEE Image, Video, and Multidimensional Signal Processing Workshop, 2017 IEEE International Workshop on Machine Learning for Signal Processing, 2018 IEEE International Symposium on Biomedical Imaging, and 2019 and 2021 International Conference on Computer Vision. He is a Senior Member of IEEE and an IEEE Computational Imaging Technical Committee member.
Zhizhen Zhao (zhizhenz@illinois.edu) received her Ph.D. degree in physics from Princeton University and her B.A. and M.Sc. degrees in physics from Trinity College, Cambridge University. She is an associate professor and William L. Everitt faculty fellow in the Department of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign (UIUC), Urbana-Champaign, IL 61801 USA. Prior to joining UIUC, she was a Courant instructor at the Courant Institute of Mathematical Sciences, New York University. She is a recipient of the Alfred P. Sloan Research Fellowship (2020–2022). Her research interests include applied and computational harmonic analysis, signal processing, and computational imaging. She is a Member of IEEE.
Raja Giryes (raja@tauex.tau.ac.il) received his Ph.D. degree from the Technion-Israel Institute of Technology. He is an associate professor in the School of Electrical Engineering at Tel Aviv University, Tel Aviv 69978, Israel. He received the European Association for Signal Processing Best Ph.D. Award, the European Research Council Starting Grant, the Maof prize for excellent young faculty (2016–2019), the VATAT scholarship for excellent postdoctoral fellows (2014–2015), the Intel Research and Excellence Award (2005, 2013), and the Excellence in Signal Processing Award from Texas Instruments (2008), and he was part of the Azrieli Fellows Program (2010–2013). He is an associate editor of IEEE Transactions on Image Processing and Elsevier’s Pattern Recognition and has organized workshops and tutorials on deep learning theory in various computer vision conferences. He is also a co-organizer of the Israel Computer Vision Day. He is a Senior Member of IEEE and has been a member of the Israeli Young Academy since 2022.
Jong Chul Ye (jong.ye@kaist.ac.kr) received his Ph.D. degree from Purdue University. He is a professor in the Graduate School of Artificial Intelligence and an adjunct professor in the Department of Bio/Brain Engineering and the Department of Mathematical Sciences at the Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Korea. Previously, he was a postdoctoral fellow at the University of Illinois at Urbana- Champaign, and a senior researcher at Philips Research and GE Global Research. He is an associate editor of IEEE Transactions on Medical Imaging, a senior editor of IEEE Signal Processing Magazine, and an executive editor of Biological Imaging. He was a guest editor for several IEEE special issues. He is a Fellow of IEEE and was the chair of the IEEE Signal Processing Society Computational Imaging Technical Committee and an IEEE Engineering in Medicine and Biology Society Distinguished Lecturer. He was a general cochair for the 2020 IEEE International Symposium on Biomedical Imaging.
[A1] R. Guo, T. Huang, M. Li, H. Zhang, and Y. C. Eldar, “Physics-embedded machine learning for electromagnetic data imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 18–31, Mar. 2023, doi: 10.1109/MSP.2022.3198805.
[A2] W.-T. Su, Y.-C. Hung, P.-J. Yu, C.-W. Lin, and S.-H. Yang, “Physics-guided terahertz computational imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 32–45, Mar. 2023, doi: 10.1109/MSP.2022.3198807.
[A3] S. Pinilla, K. V. Mishra, I. Shevkunov, M. Soltanalian, V. Katkovnik, and K. Egiazarian, “Unfolding-aided bootstrapped phase retrieval in optical imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 46–60, Mar. 2023, doi: 10.1109/MSP.2022.3214325.
[A4] J. Chen, M. Zhao, X. Wang, C. Richard, and S. Rahardja, “Integration of physics-based and data-driven models for hyperspectral image unmixing,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 61–74, Mar. 2023, doi: 10.1109/MSP.2022.3208987.
[A5] H. Arguello et al., “Deep optical coding design in computational imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 75–88, Mar. 2023, doi: 10.1109/MSP.2022.3200173.
[A6] W. Xia, H. Shan, G. Wang, and Y. Zhang, “Physics-/model-based and data-driven methods for low-dose computed tomography,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 89–100, Mar. 2023, doi: 10.1109/MSP.2022.3204407.
[A7] F. Lam, X. Peng, and Z.-P. Liang, “High-dimensional MR spatiospectral imaging by integrating physics-based modeling and data-driven machine learning,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 101–115, Mar. 2023, doi: 10.1109/MSP.2022.3203867.
[A8] Y. Zhu, J. Cheng, Z.-X. Cui, Q. Zhu, L. Ying, and D. Liang, “Physics-driven deep learning methods for fast quantitative magnetic resonance imaging,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 116–128, Mar. 2023, doi: 10.1109/MSP.2023.3236483.
[A9] Q. Yang, Z. Wang, K. Guo, C. Cai, and X. Qu, “Physics-driven synthetic data learning for biomedical magnetic resonance,” IEEE Signal Process. Mag., vol. 40, no. 2, pp. 129–140, Mar. 2023, doi: 10.1109/MSP.2022.3183809.
Digital Object Identifier 10.1109/MSP.2023.3236492