figure[Getty Images]

Hyperspectral imaging—the collection of images with spectral data in each pixel—has proved its worth in many data-gathering applications, from environmental sensing and agricultural crop monitoring to food inspection and art conservation. But can this technique aid scientists working at the opposite end of the size scale: searching for problems with tiny human cells and their components?

Few devices have passed all the approvals to gain widespread use, but researchers hope that hyperspectral imaging will follow in the footsteps of laser eye surgery and skin treatments.

The short answer is “yes,” but mainly in the research laboratory. Scientists and engineers from multiple disciplines are adapting the spectral-based hardware and software from remote sensing to biomedical applications. Few devices have passed all the approvals to gain widespread use in clinicians’ hands, but researchers hope that hyperspectral imaging will follow in the footsteps of laser eye surgery and skin treatments.

In the lab, much of the hyperspectral imaging research has been focused on two main agents of mortality: cancer and Alzheimer’s disease. Scientists have also tested the potential of hyperspectral imaging to locate ischemic intestinal tissue during surgery, to test a non-invasive scanning method for high cholesterol, to predict the development and healing of diabetic foot ulcers, and to diagnose age-related macular degeneration.

Here is a closer look at the technology for biomedical hyper- and multispectral imaging and a few potentially exciting applications.

figure

figureTop: Baowei Fei, UT Dallas, TX, USA, with a prototype of his smart surgical microscope that uses hyperspectral imaging and artificial intelligence to detect cancer. Bottom: Original tissue sample (left), current microscope method in identifying cancer cells (center) and hyperspectral imaging (right). [UT Dallas]

Adapting for biology

Surprisingly, scientists do not have a precise definition of hyperspectral imaging. Some researchers define “hyperspectral” as hundreds of wavelength bands for each pixel, while “multispectral” refers to a smaller number, perhaps 3 to 20. In another interpretation of the terminology, hyperspectral imaging uses a continuous range of spectral bands, whereas multispectral imaging uses discrete, more widely spaced bands. Some researchers lump these methods under the umbrella term “spectral imaging.”

“Where we’ve kept the wavelengths close enough together that we can construct a smooth, continuous spectral response out of that, that’s more similar to hyperspectral imaging, and we label it as that,” says Silas Leavesley, professor of chemical and biomolecular engineering at the Center for Lung Biology at the University of South Alabama, USA.

OSA Fellow Adam Wax, professor of biomedical engineering at Duke University, USA, says that hyperspectral imaging at biomedical scales can either sweep through the wavelength range for colors being delivered and then record whatever comes back, or illuminate the target with broadband light and record the discrete reflected wavelengths.

“It depends on where you want to spend your money on equipment,” Wax says. Tunable lasers tend to be expensive. For biomedical engineers who eventually want to market their devices to clinician end users, having wavelength selectivity on the detection end is a paramount consideration. Thus, the dispersive components in biomedical hyperspectral imaging systems range from lower-cost liquid-crystal tunable filters to more elaborate acousto-optic deflectors.

In a 2019 review article, biomedical researchers in the United States and Spain—led by Baowei Fei, a professor of bioengineering at the University of Texas at Dallas, USA—listed four types of acquisition modes used in hyperspectral imaging: point-scanning or “whiskbroom,” line-scanning or “pushbroom,” plane-scanning and snapshot. Detectors can be CCD cameras, CMOS-based cameras or near-infrared detectors based on mercury cadmium telluride or indium gallium arsenide.

The remote-sensing community developed the original processing algorithms for hyperspectral image sets, also known as hyperspectral data cubes, but scientists are now modifying the approaches to meet the needs of biology and medicine. In so-called supervised classification algorithms, users “train” the algorithm to recognize certain conditions based on a set of spectral signatures. Unsupervised classification algorithms divide images into clusters of pixels that share approximately the same spectral characteristics. These algorithms don’t seek to identify the nature of the material in each cluster, but they excel at delineating boundaries of spectral regions—helpful for mapping the extent of tumors.

Spectral imaging does not require the recently developed super-resolution microscopy techniques that exceed the conventional diffraction limit, says Richard M. Levenson, professor of pathology at the University of California at Davis, USA. Rather, hyperspectral and multispectral cameras can be teamed up with regular laboratory microscopes that show cells and their components, such as the nucleolus and chromatin.

figureThomas Rich (left) and Silas Leavesley. [USA Health/Bill Starling]

Leavesley and his colleagues, including South Alabama pharmacology professor Thomas C. Rich, study spectral imaging on a range of size scales, from macroscopic tissues to subcellular processes. “Think down at the microscopic level looking at cell signaling and concentrations of different molecules within a cell,” Leavesley says, “up to the macroscopic level, trying to develop types of spectral endoscopes for looking at whole tissue.”

On the one hand, the South Alabama researchers study cellular signaling systems to determine how information is encoded in these cellular response regulators. According to Rich, that requires fast sampling of dim fluorescent signals. On the other hand, the lung biology center seeks new technologies to aid in endoscopic imaging.

In one recent study, the South Alabama scientists combined the small and large imaging scales. When green fluorescent proteins came into use as tissue labels in the 1990s, the autofluorescence, or naturally occurring fluorescence, of substances such as collagen and elastin became a source of imaging noise. Unfortunately for sufferers of colon cancer, but fortunately for researchers studying the disease, some of these autofluorescent proteins and enzymes occur in different concentrations in cancerous and benign tissues.

Cancerous and benign cells have different optical properties, a fact that has helped spark interest in using hyperspectral imaging to differentiate among cells and detect tumors.

To boost the signal-to-noise ratio of these naturally occurring indicators, Leavesley, Rich and collaborators examined human colon tissue samples with an excitation scanning microscope looking for a range of short wavelengths, 360 to 530 nm, in 5-nm bands. So far, the results look promising for distinguishing normal cells from neoplasias, and perhaps the work could lead to a new type of endoscopic viewing and diagnosis tool for clinical use.

Seeking cancer cells

Cancerous and benign cells have somewhat different optical properties, a fact that has undoubtedly helped spark interest in using hyperspectral imaging to differentiate among cells and detect tumors. One area where this distinction is particularly important is the brain.

Surgery for metastatic brain cancer is a Goldilocks problem: remove too little tissue and the tumor grows back, or cut out too much tissue and leave the patient with potential neurological deficits. In search of the just-right excision, physicians have tried magnetic resonance imaging (MRI), which requires special equipment and extends the time of the surgery; ultrasound, which lacks sufficient image resolution; and fluorescent markers, which are too toxic for pediatric use.

figureHyperspectral image of colorectal adenocarcinoma cell line Caco-2, acquired with a wide-field hyperspectral microscope. Cell membranes (red) and nuclei (green) are stained with fluorescent dyes. Fluorescence from Alexa Fluor 430 has been expanded 10× for clarity. [A. Candeo et al., APL Photonics 4, 120802 (2019); CC by 4.0]

A multinational group, including Spanish and British scientists led by Gustavo M. Callicó of the University of Las Palmas de Gran Canaria, Spain, recently wrapped up an ambitious European Commission–funded project called HELICoiD, for Hyperspectral Imaging Cancer Detection. HELICoID’s goal was to combine hyperspectral imaging with classification algorithms to assist neurosurgeons in detecting brain-tumor margins in real time.

The HELICoiD researchers built a data-processing pipeline with both supervised and unsupervised classification algorithms—the former to improve the spatial coherence of the data and the latter to delineate tissue boundaries accurately. In an operating theater, the physicians working with HELICoiD captured hyperspectral images with a pushbroom-type camera in the wavelength range from 400 to 1000 nm, although they later determined that 450 to 900 nm was the most efficient range for this procedure. The algorithms generated mapped images of brain tissue in roughly one minute, far better than the half an hour needed for an MRI.

The proof-of-concept HELICoiD study produced, in the words of its authors, “the first public in-vivo hyperspectral human brain image database specifically for brain cancer detection.” Nevertheless, the physician and engineer collaborators noted several limitations of the study, including the limited number of patients (only 36) and the lack of diversity among their brain cancers (most were glioblastomas).

One team based at the Polytechnic University of Milan, Italy, recently combined hyperspectral imaging and interferometry to create a wide-field microscope system that can view fluorescently tagged cancer cells. According to the researchers, led by OSA member Cristian Manzoni, multispectral and hyperspectral imagers don’t work well with wide-field microscopes, due to long acquisition times, reduced spatial resolution or high complexity.

Rather than dispersing light in the frequency domain, the Italy-based group turned to Fourier transformation. The researchers built a small common-path interferometer based on birefringent crystals that can retard one of two polarized light beams for hundreds of cycles. Combined with a commercial microscope and CCD camera, the device captured distinct images (in the 400-to-1000-nm range) of a known colorectal adenocarcinoma cell line. Since the barium borate crystals in the interferometer can pass light with wavelengths as long as 3500 nm, the team could extend the instrument’s range into the near-infrared by switching out the detector.

Fighting the scourge of Alzheimer’s

As the association between amyloid-beta plaques in the human brain and Alzheimer’s disease has grown over the past two decades, scientists have hotly pursued diagnostic techniques to identify the devastating malady in its earliest stages. Since these protein aggregates appear in the eye’s retina, research teams have investigated various strategies for detecting retinal amyloid beta before patients start exhibiting symptoms such as dementia.

figureThe Centre for Eye Research Australia (CERA) is developing a retinal imaging strategy that could give individual patients an actual early warning of impending dementia. [Getty Images]

Peter van Wijngaarden, deputy director of the Centre for Eye Research Australia (CERA), says he was interested in developing a retinal imaging strategy that could give individual patients an actual early warning of impending dementia. “We were really struck by the fact that a number of studies have shown that these associations are useful at a population level,” he says, “but not at the individual level where it counts.” He and his associates were motivated by a 2015 University of Minnesota, USA, study that demonstrated in the lab that when amyloid beta reaches a certain aggregation stage within cells, it exhibits a shift in its spectral reflectance.

Van Wijngaarden has been working with a postdoctoral fellow, Xavier Hadoux, whose Ph.D. work involved the processing and interpretation of the massive data sets arising from agricultural hyperspectral imaging. Last year, they and other researchers from CERA and other institutes in Melbourne, Australia, reported the first use of an in vivo hyperspectral system for detecting amyloid beta in the human retina.

To test patients, van Wijngaarden, Hadoux and their colleagues built a stand that holds a camera and plastic forehead and chin rests to keep the patient’s head steady, much like patients would use during a regular eye examination.

To make each measurement, the experimental apparatus emits a brief pulse of near-infrared light to help the camera focus. Then the device illuminates the subject’s eye with a spectrum of visible light lasting a little less than a second. According to van Wijngaarden, the subject perceives the pulse as a brief flash of rainbow color, adding that “many of our participants enjoy the experience.”

One constraint on the light going into the eye: the range of wavelengths that the eye’s components will transmit. As people age, their eyes become less transparent to short wavelengths. The camera-controlling software also had to account for subjects’ microsaccades, or tiny, involuntary eye movements that occur while people fix their gaze on a stationary point. Finally, Hadoux explains, the team needed a system with fast acquisition times so that the imaging seemed acceptably short to patients.

The CERA researchers found that people who had already displayed high levels of amyloid-beta plaques on brain scans, and who also displayed mild cognitive impairment, had significantly different retinal reflectance spectra from people whose brain scans had no evidence of such plaques. One challenging part of the study, he notes, was finding subjects who test positive for the plaques either through a positron emission tomography (PET) scan or a lumbar puncture. Not all dementia patients choose to get these tests.

In 2019, the Alzheimer’s Drug Discovery Foundation awarded van Wijngaarden one of four inaugural grants from its Diagnostics Accelerator initiative to boost early detection strategies for combating the disease. The CERA deputy director is one of two retinal-study awardees, but the only one explicitly using hyperspectral imaging. Van Wijngaarden and Hadoux plan to build more hyperspectral camera systems for colleagues in other countries to use, leading up to eventual clinical trials. They hope to determine whether hyperspectral imaging can supplement or even replace PET scanning as a diagnostic tool for Alzheimer’s.

figureHyperMed Imaging’s self-contained tissue oximetry device. [HyperMed Imaging]

Commercial possibilities

Even though biomedical hyperspectral imaging is a relatively new application, some optical instrumentation companies are starting to commercialize it. For example, U.S. companies Headwall Photonics Inc. and Resonon Inc. make hyperspectral systems for life sciences. In Italy, a Polytechnic University of Milan spin-off company, Nireos, has commercialized its Fourier-transform–based hyperspectral technology into a camera suitable for biomedical imaging. Thorlabs Inc., NJ, USA, sells a benchtop hyperspectral system—including a liquid-crystal tunable filter and monochrome CCD camera—as well as the individual components. A Canadian startup company, Optina Technologies, is trying to commercialize CERA’s approach to comparing hyperspectral retinal images with PET brain scans.

One company exploring multiple purposes for commercialized hyperspectral imaging is TruTag Technologies, CA, USA. Although TruTag and its imaging business unit, HinaLea, developed their portable spectral imaging unit to read the spectra of silica powder embedded into material goods for identification purposes, the company’s vice president of engineering, OSA Fellow Alexandre Fong, says the devices, based around a Fabry-Pérot etalon, can work in biomedical applications as well.

Another company, HyperMed Imaging Inc., TN, USA, markets a self-contained tissue oximetry device called HyperView, which the U.S. Food and Drug Administration has approved for differentiating between oxygenated and deoxygenated hemoglobin in blood vessels just below the surface of the skin, especially in diabetic patients. The product aims to screen patients for peripheral artery disease and other ischemic problems that could lead to serious complications. The company calls its product proprietary but cites several studies from the past decade.

figureExtraction of blood and melanin information from hyperspectral reconstruction with RGB images captured by a smartphone camera. Red arrow: skin redness or pimples; black arrow: moles. Blood and melanin absorption maps are coded according to the color bar shown on the right. [Q. He and R. Wang, University of Washington]

Future prospects

Perhaps someday biomedical scientists won’t even use these lab-bench systems—they will simply pull their hyperspectral imagers out of their pockets. Two scientists at the University of Washington, USA, recently turned a typical smartphone, with a camera ranging from 8 to 12 million pixels, into a device that performs at least multispectral imaging. To tease spectral information out of the standard RGB phone camera, the researchers took identical pictures of a color chart with a phone and a 16-channel snapshot reflectance spectral camera. The pair then used an algorithm called a Wiener estimation matrix calculation to calibrate the phone with the spectral imager.

With 16 bands between 470 and 620 nm, the Washington pair extracted absorption spectral information on two chromophores, hemoglobin and melanin, from pictures of human skin taken with the smartphone. The scientists even measured a human pulse rate from skin spectral imaging and detected a mock vascular occlusion created by wrapping a finger with a rubber band. Such a calibrated smartphone camera could be used for routine dermatological screening or skin examinations in low-resource regions.

Perhaps someday biomedical scientists won’t even use these lab-bench systems—they will simply pull their hyperspectral imagers out of their pockets.

By 2014, observers of the field had noted three clear challenges to the field of hyperspectral imaging: real-time acquisition of high-resolution hyperspectral data sets, fast processing of these vast data sets and the establishment of a large spectral database for important molecular biomarkers.

Today, scientists have made significant progress in fast image acquisition and data processing speeds, says Fei. He believes the community still needs better spectral databases. Several groups are working on the database problem, including one at the U.S. National Institute of Standards and Technology (NIST), which is compiling a reference data set of human skin reflectance.

figureNIST researchers have gathered skin reflectance data to establish the variation found in human tissue to develop reference standards for hyperspectral imaging applications. Left: Skin as normally viewed. Right: The same images with enhanced contrast in false color to show the variability between subjects. [Catherine C. Cooksey and David W. Allen, NIST]

“The long-term storage of many large data sets is a problem,” Rich says, “but it is also a really good opportunity for groups that do data mining.”

Interest in applying spectral imaging to basic biology and clinical medicine is growing. The U.S. government’s online database of clinical trials lists 39 studies associated with the term “hyperspectral,” although not all are active. For example, one study, based in New York, USA, seeks patients suffering from age-related macular degeneration, a sight-robbing condition, for assessments using a combination of hyperspectral imaging and optical coherence tomography.

“The basic premise of multispectral [imaging] is getting more from less,” Levenson says. “We’re challenged as pathologists and clinicians getting more and more information out of smaller and smaller pieces of tissue. And so the more information you can get out of a single chunk, the better off everyone is.” Spectral imaging helps scientists better understand the spatial relationships between cells, he adds, leading to more nuanced views of cancer biology and immunology.

Rich envisions many meaningful opportunities in biomedical hyperspectral imaging for young scientists coming into the field. “We’re having developments in the technologies to do spectral filtering,” he says. “We’re having really rapid developments in sensor technology, coinciding with a lot of work using a wide variety of approaches for data mining.”


Patricia Daukantas is a freelance science writer based in Greenbelt, MD, USA.

References and Resources

For references and resources, go online: www.osa-opn.org/link/bio-hyperspectral.