A representation of a neural network provides a backdrop to a fish larva’s beating heart. [Image: Tobias Wuestefeld]
The advent of deep learning, a powerful form of machine learning, has led to rapid advancements in areas such as speech recognition, visual object recognition, genomics and drug discovery. These methods are characterized by multiple processing layers that can tease out intricate patterns and structures in very large, complex data sets.
Now, a team of European researchers has incorporated deep learning algorithms into a light-field microscope to enhance both its reconstruction speed and image quality (Nat. Methods, doi: 10.1038/s41592-021-01136-0). The results significantly extend the capabilities of light-field microscopy for whole-brain or whole-animal imaging of living specimens for biomedical research.
The need for speed
Many biological processes, like the firing of neurons or the movement of red blood cells, happen extremely fast. To capture such dynamics, imaging techniques need to have temporal resolution on the scale of milliseconds or better. However, recording large samples at high speed in three dimensions has been a recurring challenge in microscopy.
Light-field microscopy stands out from other approaches in this respect since it can instantaneously capture 3D spatial information in a single camera frame. Unfortunately, image reconstruction with this method on a standard computer can take hours or even days with disappointing, low-quality results.
“While light-field microscopy can acquire high-speed, 3D imaging data, reconstructing the actual images from the raw data takes a long time and often produces unwanted artifacts,” said Robert Prevedel, study author and group leader at the European Molecular Biology Laboratory (EMBL). “Therefore, we aimed to speed-up the reconstruction process and to improve the quality of the 3D images by incorporating artificial intelligence into the computational workflow of the microscope.”
Best of both worlds
Prevedel and his colleagues created a deep-learning algorithm—specifically, a convolutional neural network called HyLFM-Net—to handle light-field data processing and image reconstruction. To boost its performance, HyLFM-Net is fed high-resolution image data gathered by light-sheet microscopy for training and validation. Light-sheet microscopy is a different imaging technique that focuses on capturing a single 2D plane at a time, but in greater detail.
By combining the two techniques, the researchers created a hybrid form of light-field microscopy that delivers high-quality 3D reconstructions in near-real time. As a demonstration, they were able to image the beating heart and whole-brain neuronal activity of live fish at rates up to 100 Hz.
Improved reliability
Another unique feature of their work is that the light-sheet microscopy component is constantly feeding new high-resolution images to HyLFM-Net, meaning that the algorithm’s performance is continuously checked and updated. This aspect created improved reconstruction reliability on top of a much higher rate of data analysis.
“If you have hundreds or thousands of samples that need to be imaged in 3D over time, one would normally wait forever—weeks or years depending on the amount—for ‘normal’ image reconstructions, while our technique can do it in a few seconds or hours,” said Prevedel.