1.

Introduction

Blood oxygen saturation (${\mathrm{SO}}_2$) is defined as the fraction of oxygen-saturated hemoglobin. Arterial ${\mathrm{SO}}_2$ can be monitored through pulse oximetry (PO) and provides information on how well the lungs oxygenate blood.¹ Peripherally, the lack of significant blood pulsations makes PO insensitive to microvascular ${\mathrm{SO}}_2$. Instead, transcutaneous oximetry (${\mathrm{tcpO}}_2$) can be used to measure the locally available free oxygen that has diffused from the microvascular blood.² The ${\mathrm{tcpO}}_2$ is a technique that only measures slow changes in one point. Spectroscopic optical techniques, including near-infrared spectroscopy³ (NIRS) and diffuse reflectance spectroscopy^4,5 (DRS), allow for the fast and direct assessment of local ${\mathrm{SO}}_2$ levels in the microcirculation and smaller vessels. Due to a superficial sampling depth for DRS in the visible wavelength range, compared with NIRS, DRS mainly monitors capillary and upper dermis microvascular vessels, whereas NIRS includes deeper and larger vessels when measuring skin tissue.⁶

Microvascular networks are spatially heterogeneous and complex structures of arterioles, capillaries, and venules connecting the arterial and venous systems. Blood flow in the network varies temporally due to local regulatory and metabolic requirements of surrounding tissues and cells.⁷ Regulation involves adjusting organ perfusion to facilitate the oxygen exchange and carbon dioxide removal; the transport of hormones, nutrients, and drugs; and the immune response, among other functions. The local microvascular ${\mathrm{SO}}_2$ is hence a marker for tissue viability and could be used for studying different peripheral vascular diseases.^8–11

Braverman et al.¹² and Tenland et al.¹³ showed spatial heterogeneity in the microvasculature in healthy skin by studying variations in measured microcirculatory perfusion using single-point methods. The exact measurement position may, therefore, have a significant impact when determining skin microcirculatory disorders and regulation. Hence, imaging methods such as laser Doppler perfusion imaging,¹⁴ hyperspectral imaging (HSI), or multispectral imaging (MSI)¹⁵ may be preferred. Previous studies have shown that HSI and MSI can aid the diagnosis of diseases related to arterial occlusions such as peripheral arterial disease,^16,17 diabetic foot,¹⁸ and wound diagnostics.^19,20

Spectral images are often acquired by scanning through either the spectral or spatial domain, leading to long acquisition times and/or low resolution, hampering the widespread adoption of HSI and MSI.²¹ This includes push-broom setups,^6,22 tunable Fabry–Pérot filters,²³ or setups that cycle through wavelengths using filter wheels or multiple illumination bands.²⁴ Using snap-shot sensors^25–28 allows for much faster acquisition rates.

Analysis of HSI and MSI data is often slow when using advanced iterative algorithms such as an inverse diffusion model in a two-layer skin model⁶ or searches in multi-dimensional Monte Carlo (MC)-simulated reflectance look-up tables.²⁵ Fast solutions, based on Beer–Lambert’s law in selected wavelength regions in single-layer skin models, have been proposed.²² MC simulations, however, give a more accurate solution to the photon transport equation in comparison with diffusion approximations and models based on Beer–Lambert’s law. The inverse MC method is considered the gold standard and thus should be the preferred choice for a precise analysis of DRS, HSI, and MSI data.

Recently, artificial neural networks (ANNs) trained on MC-simulated data from realistic multilayer skin models^23,29 show that MC and ANN can be combined for a fast analysis of DRS and HSI data. Ewerlöf et al.²⁶ also showed how ANN trained on in-vivo data from a reference device can be used for a precise and fast analysis of MSI snapshot images. This approach circumvents the need for handling the complex spectral response found in snapshot sensors, characterized by broad spectral bands or bands with multiple peaks. Unfortunately, it introduces the need for acquiring training data from a large representative population at different ${\mathrm{SO}}_2$ levels for every MSI snapshot camera as the sensor characteristics differ between each unique sample.

This study aims to develop a framework for training an ANN to estimate ${\mathrm{SO}}_2$ from snapshot MSI data with a precision comparable to inverse MC. The framework includes developing a model that closely mimics both tissue reflectance and instrumental characteristics for the generation of synthetic high-quality training data. An integral part of modeling is to account for variations in model characteristics to enable robust ${\mathrm{SO}}_2$ estimations insensitive to, e.g., tissue melanin, tissue scattering, instrumental noise, and instrumental drift, allowing for accurate real-time imaging of microcirculatory ${\mathrm{SO}}_2$.

2.

Material and Methods

All images were acquired using an MSI snapshot camera (MQ022HG-IM-SM4X4-VIS, XIMEA GmbH, Münster, Germany) with 16 different optical bands in the 470 to 650 nm wavelength range. The camera had a complementary metal-oxide semiconductor (CMOS) imaging sensor with a native resolution of $2048\times 1088\text{pixels}$, where each pixel had a Fabry–Pérot bandpass filter deposited on top of it. The 16 unique bandpass filters were arranged in a recurring $4\times 4$ mosaic pattern, resulting in an MSI hypercube ($16\times 512\times 272$). The camera was connected to a computer via USB3, allowing for a maximal theoretical frame rate of 170 fps. In our study, the actual frame rate was limited to 4.2 fps due to the chosen exposure time and on-the-fly averaging of multiple consecutive images.

The camera was equipped with a 16-mm c-mount lens (TECHSPEC, Edmund Optics, Barrington, New Jersey, United States) with the aperture size set to f/1.6. A long-pass (LP) filter with a cut-off wavelength of 470 nm (LP470 stablEDGE, Midwest Optical Systems, Palatine, Illinois, United States) was attached to the lens to suppress the dominant blue peak found in fluorescent white light-emitting diodes (LEDs). The spectral response for each of the 16 bands is influenced by, e.g., the attached optics and lens aperture size. Therefore, to complement the spectral responses supplied by the manufacturer, an in-house calibration was performed with the lens and LP filter attached. In addition, the spectral characteristics of the 8 W white LED ring light (R130, Smart Vision Lights, Norton Shores, Michigan, United States) and polarizing filters used in the in-vivo measurements were determined using a calibrated spectrometer (AvaSpec-2048L-USB2, calibrated using AvaLight-HAL-CAL-Mini; Avantes B.V., Apeldoorn, The Netherlands). The complete camera and LED setup was further characterized by repeated diffuse reflectance measurements from a white reference Spectralon target with a 99% nominal reflectance (Labsphere Inc., North Sutton, New Hampshire, United States). These measurements were used to describe temporal noise in the pixel intensity and to white-normalize all acquired in-vivo MSI data.

An optical model describing the camera system was combined with MC simulations of a tissue model describing the spectral reflectance of skin tissue. This combined model was used both in an inverse MC algorithm and for generating training data for an ANN algorithm trained to assess blood ${\mathrm{SO}}_2$ in skin tissue. The inverse MC and ANN algorithms were evaluated on in-vivo measured data from a healthy subject during an occlusion-release provocation.

2.1.

Characterization of Sensor Spectral Response

The spectral response of the 16 different wavelength bands in the MSI camera was characterized using the calibration setup in Fig. 1. The setup included a white LED spotlight (SXA30, Smart Vision Lights) mounted in front of a liquid crystal tunable filter (LCTF; VariSpec VIS-7, CRi Inc., McLean, Virginia, United States). A standard optical diffusor was placed after the LCTF to further mix and diffuse the filtered light. The light was then diffusely reflected by a white reference Spectralon tile and captured simultaneously by the MSI camera and an optical fiber connected to the calibrated spectrometer. The MSI camera included a 16-mm lens and LP470 filter.

Fig. 1

MSI camera calibration setup with a calibrated spectrometer and an LCTF-based tunable light source. The setup was used for quantifying the spectral response of the 16 different wavelength bands.

The LCTF, with a 7 nm full width at half maximum (FWHM) transmission peak, was computer controllable, allowing for an automated scanning from 400 to 700 nm in steps of 1 nm. At each wavelength, MSI images and calibrated spectra were acquired. Dark MSI images and spectra were additionally recorded at every 10th wavelength.

Data were processed by first subtracting the dark level from the acquired MSI images and the high-resolution spectra acquired by the calibrated spectrometer. The high-resolution spectra were then smoothed with a 1 nm wide moving average filter, to avoid aliasing, and down-sampled to a 1 nm resolution between 400 and 700 nm. The MSI data were spatially averaged using a region of interest (ROI) that included all pixels with an intensity of more than 85% of the maximum intensity in the wavelength-averaged image. This resulted in a centrally positioned fixed-size ROI that included 11% of the sensor pixels.

The intensity detected by the MSI camera in the calibration setup is modeled as

Eq. (1)

$$I_{\mathrm{C}}[n,m]=\sum _{\lambda =400}^{700}{r}_n(\lambda )R_m(\lambda ),$$

where $n$ marks the spectral band for the MSI sensor, $r_n(\lambda )$ is the unknown response for each spectral band, and $R_m(\lambda )$ is the reflected irradiance as detected by the calibrated spectrometer while having the LCTF filter tuned to wavelength setting $m$. As a result of the 7 nm FWHM transmission peak of the LCTF, the spectral response is not given directly by the calibration dataset. Hence, the spectral response was estimated using a non-linear optimization algorithm to find the $r_n$ that minimizes the difference between the modeled and measured MSI intensities for all LCTF settings. Rapid changes in the estimated spectral response between consecutive wavelengths were additionally penalized to make the solution less sensitive to measurement noise.

The applied penalty function is given by

Eq. (2)

$${\chi }_{r_n}=[\frac{I[n,m]}{{⟨I[n,m]⟩}_{n,m}}-I_{\mathrm{C}}[n,m],0.2\mathrm{\Delta }{r}_n(\lambda )],$$

where $I[n,m]$ is the intensity detected by the MSI camera in band $n$ using LCTF wavelength $m$. The normalization with the average overall detected intensity ${⟨I[n,m]⟩}_{n,m}$ makes the inverse solution insensitive to, e.g., overall lamp intensity, camera exposure time, and gain settings. No such normalization factor is needed for the modeled intensity $I_{\mathrm{C}}[n,m]$ as this factor is instead handled by the overall magnitude of $r_n(\lambda )$. The additional term $\mathrm{\Delta }{r}_n(\lambda )$ in the penalty function is given by

Eq. (3)

$$\mathrm{\Delta }{r}_n(\lambda )=\frac{r_n({\lambda }_{i+1})-r_n({\lambda }_i)}{0.5r_n({\lambda }_i)+0.5r_n({\lambda }_{i+1})+{⟨r_n(\lambda )⟩}_{\lambda }}.$$

The $\mathrm{\Delta }{r}_n$ penalty term was designed as an amplitude-normalized derivative of the estimated sensor response. The normalization is equally balanced between the local and the global average amplitude. The local amplitude normalization improves spectral characteristics for low amplitudes, and the global amplitude normalization adds stability by avoiding zero division for a wavelength with zero response. The optimal response $r_n^{*}(\lambda )$ with the minimal penalty is found using a nonlinear least-square fitting algorithm (lsqnonlin in Matlab 2021a, The Mathworks Inc., Natick, Massachusetts, United States) that solved

Eq. (4)

$$\underset{r_n}{\min }{\Vert {\chi }_{r_n}\Vert }_2^2.$$

The optimal sensor response $r_n^{*}(\lambda )$ (a.k.a. in-house sensor response) and the sensor response supplied by the camera manufacturer were validated by comparing white-normalized measured and modeled MSI intensities from a tissue-like wide-band phantom. The validation data were acquired by replacing the LCTF with a skin-like transparent color filter (Roscolux #4615, Rosco Laboratories Inc., Stamford, Connecticut, United States). Spectral characteristics of the 470 nm LP filter were added to the spectral response in Eq. (1) when evaluating modeled intensities based on the sensor response data supplied by the manufacturer.

2.2.

Modelling In-Vivo Detected MSI Intensities

Intensities detected by an MSI snapshot camera depend not only on the unique and complex response of each wavelength band but also on the LED irradiance, transmission of additional optical components, and reflectance of the target. Together, these effects contribute to the modeled detected intensity $I_{\mathrm{T}}[n]$ in wavelength band $n$ as outlined in the schematic overview in Fig. 2 and described by

Eq. (5)

$$I_{\mathrm{T}}[n]=k_n\sum _{\lambda =400}^{700}{r}_n^{*}(\lambda )F(\lambda )L(\lambda )T(\lambda ),$$

where $F(\lambda )$ is the spectral transmission of additional optical components (e.g., polarizing filter) not included in $r_n^{*}$, $L(\lambda )$ is the LED emission spectrum, and $T(\lambda )$ is the spectral reflectance of the target.^25,28 The detected intensity in each pixel is also dependent on, e.g., local variations in the on-chip filter mosaic, setting of the lens, and analog-to-digital converters. The overall amplitude of these factors may vary spatially over the sensor and hence affect each band/pixel slightly differently. The local pixel-amplification effect is modeled by the coefficient $k_n$.

Fig. 2

Schematic overview of how MSI intensities are modeled from the sensor response, lamp intensity, and MC-simulated reflectance using a two-layered skin tissue model. The modeled MSI intensities are used in the slow iterative inverse MC analysis (purple annotations) and the ANN training (green annotations), with variations and noise added, targeting estimation of the ${\mathrm{SO}}_2$ level. The blue box and lower blue arrow mark how MSI measurements are analyzed at high speed using a pre-trained ANN algorithm without needing any additional modeling of MSI intensities (in contrast to inverse MC).

To effectively cancel out any influence from a $k_n$ that varies spatially over the imaging sensor, all MSI data were normalized with a hypercube measured on a white reference target. The white-normalized MSI spectra were also normalized with the local spectral-averaged intensity (i.e., average overall wavelength bands) to further reduce any dependency on, e.g., uneven emission intensity, target distance, and white calibration distance. For modeled intensities, this results in a normalized detected intensity that is calculated as

Eq. (6)

$$N[n]=\frac{I_{\mathrm{T}}[n]}{I_{\mathrm{W}}[n]}/{⟨\frac{I_{\mathrm{T}}[n]}{I_{\mathrm{W}}[n]}⟩}_n,$$

where ${⟨\dots ⟩}_n$ denotes the average over all spectral bands and $I_{\mathrm{W}}[n]$ is the MSI intensity from a white reference target modeled with $T(\lambda )=1$.

2.4.

Generating Data for ANN Training

The measured MSI spectra from tissue can be modeled by combining the MC tissue model and the MSI model, as outlined in the schematic overview in Fig. 2. However, these models do not fully include variations related to hardware imperfection. To synthesize realistic ANN training data, valid for all pixels on the sensor, such variations need to be added. We identified three different imperfections that significantly influence the MSI spectrum. This includes temporal drift in the LED emission spectrum, angular variations in the LED emission spectrum, and temporal noise-related variation in the detected intensity. Adding these three terms expands the modeled MSI spectrum in Eq. (5) to

Eq. (12)

$$I_{\mathrm{T},\mathrm{ANN}}[n]=k_n(1+{ϵ}_{t,n})\sum _{\lambda =400}^{700}{r}_n^{*}(\lambda )F(\lambda )L_{\mathrm{ANN}}(\lambda )T(\lambda ),$$

where

Eq. (13)

$$L_{\mathrm{ANN}}(\lambda )=L(\lambda )(1+{\delta }_t(\lambda ))(1+{\delta }_{\theta }(\lambda )).$$

In these equations, $L(\lambda )$ is the stable emission spectrum of the LED at a 0-deg emission angle, ${ϵ}_{t,n}$ describes the temporal noise-related intensity fluctuation in each band $n$, ${\delta }_t(\lambda )$ relates to temporal drift in the emission spectrum, and ${\delta }_{\theta }(\lambda )$ adds emission-angle-dependent variations to the emission spectrum.

Changes over time in the factor ${\delta }_t(\lambda )$ were measured for the LED using a calibrated spectrometer at a 0 deg emission angle during a 60 min warm-up period. After this period, no significant drift was found, and the stable LED emission spectrum was determined as $L(\lambda )=L_{t=60\min }(\lambda )$. During the warm-up period, all ${\delta }_t(\lambda )$ were found to scale linearly to the maximum deviation as

Eq. (14)

$${\delta }_{t,\max }(\lambda )=\frac{L_{t=0\min }(\lambda )}{L_{t=60\min }(\lambda )}-1,$$

where $L_{t=0\min }(\lambda )$ is the emission spectrum measured during the first minute of warm-up.

Variations in the factor ${\delta }_{\theta }(\lambda )$ were measured at emission angles between 0 and 22 deg, matching the maximal angle of view for the camera setup. All ${\delta }_t(\lambda )$ were found to scale linearly to the maximum deviation as

Eq. (15)

$${\delta }_{\theta ,\max }(\lambda )=\frac{L_{\theta =22\mathrm{deg}}(\lambda )}{L_{\theta =0\mathrm{deg}}(\lambda )}-1,$$

where $L_{\theta =0\mathrm{deg}}(\lambda )$ is the stable emission spectrum measured at a 0 deg emission angle and $L_{\theta =22\mathrm{deg}}(\lambda )$ is the stable emission spectrum measured at a 22 deg emission angle.

With these linear relations, Eq. (13) is rewritten as

Eq. (16)

$$L_{\mathrm{ANN}}(\lambda )=L(\lambda )(1+q_t{\delta }_{t,\max }(\lambda ))(1+q_{\theta }{\delta }_{\theta ,\max }(\lambda )),$$

where $q_t\in [\mathrm{0,1}]$ and $q_{\theta }\in [\mathrm{0,1}]$. The spectral components for modeling the lamp emission spectrum in Eq. (16) are shown in Figs. 3 and. 4.

Fig. 3

Normalized stable lamp emission spectrum measured at a 0 deg emission angle after a 60 min heat-up period.

Fig. 4

Maximal temporal (${\delta }_{t,\max }(\lambda )$) and angular (${\delta }_{\theta ,\max }(\lambda )$) variations in the lamp emission spectrum.

The temporal detector noise ${ϵ}_{t,n}$ depends on the light intensity quantified by each pixel. The noise is assumed to be Gaussian distributed with a variance that increases with intensity. Repeated white calibration measurements over time with the MSI camera, $I_{\mathrm{W}}[t,n]$, at different LED light intensity levels, support the assumption of a Gaussian distribution. It also shows that the temporal noise variance increases linearly with intensity (Fig. 5), and hence, the detector noise is modeled by

Eq. (17)

$$\mathrm{var}({ϵ}_{t,n})=a_n+I[n]b_n,$$

where $a_n$ and $b_n$ are coefficients determined from the calibration measurement and ${⟨{ϵ}_{t,n}⟩}_t=0$.

Fig. 5

Variance of temporal detector noise for each of the 16 different bands. The intensity is normalized with the maximal detectable intensity $I_{\max }$ given by the camera analog-to-digital converter resolution.

Having established a model that accounts for tissue and hardware variations, including noise, enables the synthetization of realistic MSI spectra that can be used for training an ANN to estimate blood ${\mathrm{SO}}_2$. The synthetization of a single MSI spectrum is done as follows:

• 1. generate an MC-simulated tissue reflectance spectrum $T(\lambda )$ using a randomized set of tissue model parameters that comply with the expected parameter distribution found in real skin tissue (Fig. 6)*

• 2. generate an emission spectrum $L_{\mathrm{ANN}}(\lambda )$, where $q_t$ and $q_{\theta }$ are random numbers uniformly distributed at $[\mathrm{0,1}]$

• 3. randomize an average detected MSI intensity $R_I$ from a normal distribution with a mean of half the maximal detectable intensity and a standard deviation of 20% of the maximal detectable intensity

• 4. calculate the intensity-normalized and scaled spectrum $I[n]=R_I{I}_{\mathrm{T},\mathrm{ANN}}[n]/{⟨I_{\mathrm{T},\mathrm{ANN}}[n]⟩}_n$ assuming no temporal detector noise (i.e., ${ϵ}_{t,n}=0$)

• 5. generate a detector noise factor ${ϵ}_{t,n}$ with a variance according to Eq. (17) and Fig. 5.

• 6. generate a synthetic MSI spectrum according to Eq. (12) with the detector noise being added.

  • * Please note that it is the amount of melanin, i.e., the product of the melanin fraction and the epidermis thickness, rather than the melanin fraction alone, that is randomized. The actual melanin fraction is then calculated from the melanin amount divided by the epidermis thickness.

Fig. 6

Distribution of the tissue model parameters for the ANN training set.

The white normalization of measured MSI spectra was also accounted for in the models by normalizing the synthetic MSI spectra from skin tissue with synthetic white MSI spectra $I_{\mathrm{W},\mathrm{ANN}}[n]$. Synthetic white MSI spectra were generated as outlined above, except for using $T(\lambda )=1$ instead of an MC-generated tissue reflectance. Finally, the intensity normalization is calculated as

Eq. (18)

$$N_{\mathrm{ANN}}[n]=\frac{I_{\mathrm{T},\mathrm{ANN}}[n]}{I_{\mathrm{W},\mathrm{ANN}}[n]}/{⟨\frac{I_{\mathrm{T},\mathrm{ANN}}[n]}{I_{\mathrm{W},\mathrm{ANN}}[n]}⟩}_n,$$

where ${⟨\dots ⟩}_n$ marks the average over all spectral bands. This effectively removes any dependency on the absolute magnitude of the MSI spectra.

3.

Results

Data from the calibration setup show that the in-house optimal sensor response $r_n^{*}(\lambda )$ deviates from the manufacturer-supplied response by 3.4% to 9.6% (mean-absolute-error of max-normalized responses). Two representative examples of responses are shown in Fig. 7.

Fig. 7

Representative examples of in-house and manufacturer-specified spectral responses for two different spectral bands. In panel (a), a typical widening and shift toward shorter wavelengths for the in-house spectral response is observed, and in panel (b), the worst matching case, a dual-peak characteristic in which the prominent peak changes position, is observed.

The validation measurement using a tissue-like color filter phantom shows that the in-house spectral responses can model the white-normalized measured intensities more accurately compared with the responses given by the manufacturer. The result, shown in Fig. 8, demonstrates a 4.3 times lower mean-absolute-percentage-error (mape) for the in-house spectral responses.

Fig. 8

Phantom validation measurement comparing the in-house estimated spectral response with the one supplied by the manufacturer.

The calibrated and normalized in-vivo MSI data from the finger occlusion experiment were analyzed by first selecting an ROI with a size of $5\times 5\text{pixels}$ placed on the middle finger distal to the cuff position (Fig. 13). The ROI-averaged spectral data were then analyzed with inverse MC and the trained ANNs. The inverse MC analysis shows that the in-house spectral response yields significantly lower fitting errors (Fig. 9), with a time-averaged mape of 0.24% compared with a mape of 2.5% when using the manufacturer response. The time-resolved ROI data also show that the spectral response has a significant effect on the estimated level of blood ${\mathrm{SO}}_2$ (Fig. 10). The manufacturer response resulted in a consistently lower blood ${\mathrm{SO}}_2$ with a mean absolute difference ($\text{mad}={⟨|s_{\text{in-}\text{house}}-s_{\text{manufacturer}}|⟩}_t$) of 13.0%, reaching an ${\mathrm{SO}}_2$ level of $-13.7\%$ during the last 10 s of occlusion. The equivalent level for the in-house response was $-5.4\%$.

Fig. 9

Example showing (a) measured and modeled normalized intensities and (b) the relative difference between measured and modeled intensities. The modeled data originate from an inverse MC fit to a single measured MSI spectrum during baseline. The measured data are ROI-averaged and analyzed with both the in-house and the manufacturer sensor responses.

Fig. 10

Inverse MC estimated blood ${\mathrm{SO}}_2$ during an occlusion-release provocation. ROI-averaged MSI data are analyzed using both the in-house and the manufacturer sensor responses.

All ANNs from the repeated training using the in-house spectral response showed great consistency when applied to the in-vivo ROI data, with a time-averaged standard deviation in estimated blood ${\mathrm{SO}}_2$ of 1.1%-units to 1.8%-units for the different network sizes. The equivalent standard deviation for the ANNs trained with the manufacturer response was 5.9%-units to 60.0%-units, manifested as an almost random behavior with widely different time traces of estimated blood ${\mathrm{SO}}_2$ for the different repeatedly trained networks. The random behavior and standard deviation increased with the network size. Due to this inconsistency, no further meaningful comparison using the manufacturer response can be made as the results will strongly and randomly depend on which repetition of ANN training is selected. An example of ${\mathrm{SO}}_2$ estimations from 10 repeatedly trained networks, all having a hidden layer size of 15 nodes, is presented in Fig. 11(a) (manufacturer response) and Fig. 11(b) (in-house response).

Fig. 11

Example of ${\mathrm{SO}}_2$ estimations from 10 repeatedly trained networks, all having the same configuration with a hidden layer size of 15 nodes. The training data are generated with (a) the manufacturer response, with a time-averaged standard deviation of 38.2%-units, and (b) the in-house response, with a time-averaged standard deviation of 1.1%-units.

ANN training performance (mean-square error evaluated with simulated test data) using the in-house response initially improved when increasing the number of nodes in the hidden layer. However, the improvement between a hidden layer size of 15 and 20 nodes was only marginal. Applying the ANN with the best test performance trained with the in-house response using a network size of 15 nodes demonstrates a close match to the time-trace of inverse MC estimated blood ${\mathrm{SO}}_2$ (Fig. 12). The mean absolute difference (mad) between inverse MC and ANN-estimated ${\mathrm{SO}}_2$ was 1.3%-units, and the coefficient of correlation $R$ was 0.999. The only major performance difference between the two methods occurs during the initial reperfusion phase when ${\mathrm{SO}}_2$ rises quickly to its peak value. During this phase, the inverse MC algorithm fails to fully capture the rapid change, most likely due to stopping the inverse solver before finding the global optimum.

Fig. 12

Panel (a) shows time traces of inverse MC- and ANN-estimated blood ${\mathrm{SO}}_2$ in the ROI. Panel (b) shows an absolute error between the two estimates.

ANN analysis of a single MSI cube, including preprocessing with dark correction, demosaicing, white normalization, and intensity normalization, took $\sim 0.11\mathrm{s}$ using a GPU implementation in Matlab R2021a (The MathWorks Inc.) and a GeForce GTX 1070 (Nvidia Corporation, Santa Clara, California, United States). The resulting blood ${\mathrm{SO}}_2$ images are shown in Video 1 and in Fig. 13, with representative still images of the occluded (at time 350 s) and a reperfused (at time 370 s) middle finger from the in-vivo provocation experiment. The images and video display the expected occlusion-release pattern with a close-to-zero ${\mathrm{SO}}_2$ at the end occlusion and $\sim 85\%$ ${\mathrm{SO}}_2$ at the reperfusion peak. No apparent curvature-dependent effects can be seen, indicating that the demosaicing algorithm and the tissue model account for any such effects.

Fig. 13

Examples of blood oxygen saturation still images from the ANN analysis showing an occluded (a) and reperfused (b) middle finger. The 5x5 pixel ROI, placed on the middle finger distal to the occlusion cuff, is marked in both still images.

4.

Discussion

We have shown that data from a snap-shot MSI camera with complex spectral characteristics can image ${\mathrm{SO}}_2$ in real time using an ANN trained on synthetic data. It is, however, critically important to apply a detailed and precise optical modeling of both the MSI camera system and the skin tissue.

The in-house spectral response, measured with optical components mounted to the MSI camera, and the response supplied by the manufacturer showed significant differences. In general, the in-house measured responses have peaks that are broader and shifted a few nanometers toward a shorter wavelength, as shown in Fig. 7. This effect is explained by the large lens aperture (f/1.6) used in this study, resulting in light impinging on the Fabry–Pérot filter deposited on top of the sensor at a range of different angles in contrast to the optimal orthogonal incidence.³⁴ A smaller lens aperture could partly address this but would instead result in a lower signal-to-noise ratio. Preliminary measurements with a more recent version of the snapshot MSI camera indicate a better resemblance between the measured sensor response and the response specified by the manufacturer.

Evaluating differences in sensor response using a color phantom (Fig. 8) indicates that the manufacturer response is less accurate, suggesting that it is not fully valid for the lens and aperture setting used in our measurements. When applied to in-vivo data, the response difference is manifested as a large inverse MC fitting error when using the manufacturer response, as shown in Fig. 9. It is inevitable that this model error will lead to erroneous estimations of ${\mathrm{SO}}_2$. This can be seen in Fig. 10, for example, where ${\mathrm{SO}}_2$ assessed with inverse MC based on the manufacturer response deviates considerably from the one based on the in-house response.

Assessing ${\mathrm{SO}}_2$ with ANN results in standard deviations of up to 60%-units between ANNs that are repeatedly trained using the manufacturer response and random initialization. Using the in-house measured response drastically decreases this standard deviation to around 1%-units regardless of the network size. This shows that the ANN approach is much more sensitive to imperfections in the modeling of the sensor response. Previous studies have also shown that DRS training data must include realistic measurement noise and drift for accurate ANN results.²⁹ For the MSI camera system, characterization and modeling of spatial and temporal variations in the light source emission spectrum and detector noise are required. The low standard deviation between outputs of repeatedly trained ANNs and the close match between inverse MC and ANN support the validity of these noise models.

Our results emphasize the need for an accurate calibration of the MSI camera response using the same optics, lens settings, and optical filters that are used in in-vivo measurements. The spectral response is a unique feature for each MSI sensor, which means that each individual camera must be characterized. Hahn et al.³⁵ previously reported that this type of snapshot sensor may also have band-dependent variations in the sensor response across the detector area. Our work did not include any quantitative evaluation of this effect. The processed ${\mathrm{SO}}_2$ images from the in-vivo measurement showed no apparent major variation that could have contributed to this effect. This can be explained by the normalization with a white hyper cube, which effectively removes spatial variations in the response magnitude for each band.

Analysis of in-vivo measured data shows low fitting errors for the inverse MC algorithm and low standard deviations between repeatedly trained ANNs when using the in-house sensor response, as seen in Figs. 9 and 11. This strongly indicates that the applied tissue model is accurate and detailed enough to mimic the optical reflectance from skin tissue using MC simulations. The tissue model itself does not contain any limitation to the number of included absorbers or scatterers but is constrained by only including melanin and hemoglobin as absorbers, which seems sufficient for skin tissue, at least in the example presented in this work. Limitations in the model parameter space need, however, to be introduced when using the tissue model for inverse MC analysis as the algorithm relies on a pre-simulated set of simulated intensities. Similarly, when used for generating synthetic reflectance data for ANN training, all physiologically reasonable parameter variations must be covered. If not, the ANN algorithm is likely to perform poorly when used on tissue types that are not included in the model parameter space of the training data.

The overall behavior of ${\mathrm{SO}}_2$ in the in-vivo experiment (Fig. 12 and Video 1) follows the expected occlusion-reperfusion pattern when analyzing spectral data in the visible 500 to 600 nm wavelength range.^6,36 The completely deoxygenized hemoglobin found after a prolonged occlusion can be explained by the superficial penetration depth for visible light, in which only blood in superficial microvascular vessels where oxygen is allowed to diffuse and metabolize in the surrounding tissue is sampled. This contrasts with techniques in which near-infrared wavelengths with a higher penetration depth are analyzed, resulting in elevated ${\mathrm{SO}}_2$ levels due to the sampling of blood in larger transport vessels.⁶

For transparency, we chose not to limit the estimated ${\mathrm{SO}}_2$ level to the 0% to 100% range. At the end occlusion, our in-vivo example shows an ${\mathrm{SO}}_2$ level of $\sim -5\%$ for both the inverse MC and ANN analyses in the selected $5\times 5$ ROI. This is, of course, not physiologically possible. Increasing the ROI size to $25\times 25\text{pixels}$ results in an ${\mathrm{SO}}_2$ level of $-1.6\%$ for the ANN analysis, indicating that some spatial noise is present, which, to some extent, causes the negative ${\mathrm{SO}}_2$ level. The negative ${\mathrm{SO}}_2$ values can also be explained by model errors, such as missing chromophores, and small calibration errors in, e.g., the white calibration of the MSI camera. The irradiance calibration is performed with a white reference target that has a very different reflectance compared with skin tissue. This can potentially introduce errors if the detected intensity response is not perfectly linear to the backscattered irradiance. With a well-known reference target mimicking the skin tissue, the effect of such non-linearities could be minimized. This is a strategy that has been successfully used in spatial-frequency domain imaging to minimize the impact of system imperfections.³⁷

In a previous study with an optical fiber probe-based system, we showed that the type of MC model used in this study can be expanded to fully explain the diffuse reflectance spectra from a wide range of skin tissue types.³⁸ In this work, however, we did not thoroughly explore what tissue model complexity is needed to produce algorithms capable of accurate modeling of MSI reflectance of different skin tissue types. Our in-vivo example has, however, uncovered (data not shown) that a single-layer tissue model or a model lacking melanin or vessel packaging effect cannot be fitted accurately to the measured data. Another limitation is the structure of the ANN, which has not been fully evaluated. The risk for overtraining certain parts or properties of the spectrum increases with a larger network size, and the limited input training data (16 spectral bands) should reduce the need for more complex or deep ANNs.²⁶ We observe comparable results for ANNs trained with fewer nodes in the hidden layer. To further explore what tissue model complexity and parameter space is needed and how the ANN should be designed for a more generalized inverse algorithm, additional in-vivo measurements covering a range of different skin types are required. Adding a reference method to such a study would also aid in determining how well the technique works for different skin tissue types.

In a previous study,²⁶ we used the same type of MSI camera and illumination setup but the ANNs were instead trained targeting ${\mathrm{SO}}_2$ values captured in vivo on the volar side of the forearm in 20 subjects (Fitzpatrick skin types I to III) using a reference probe-based technique. A leave-one-out analysis (i.e., training on 19 subjects and evaluating on the non-included subject) showed that the MSI ${\mathrm{SO}}_2$ estimation at end-occlusion was on par with (5%-units mean absolute error) the ${\mathrm{SO}}_2$ level measured with the reference technique. Those results show that ${\mathrm{SO}}_2$ can be estimated from MSI data for skin types I to III using an ANN-based approach if hardware and tissue can be modeled accurately. To further demonstrate the accuracy of the proposed method, a validation study using phantoms with a known ${\mathrm{SO}}_2$ level would be desirable.

Our ANN approach, in which ${\mathrm{SO}}_2$ is directly estimated, can analyze a single hypercube in $\sim 0.1\mathrm{s}$. This is much faster than inverse MC, which typically takes up to 1 h to analyze a comparable-sized dataset.²⁵ The 0.1 s analysis time demonstrates that our ANN approach, paired with a snapshot MSI camera, is fast enough for real-time imaging of ${\mathrm{SO}}_2$ dynamics at several frames per second. This frame rate allows us to capture most, if not all, physiological variations in ${\mathrm{SO}}_2$.

There are several clinical situations in which the utilization of HSI and MSI techniques is emerging. During free flap surgery, HSI at 16 to 28 h post-operatively utilizing a combination of tissue ${\mathrm{SO}}_2$ and a perfusion proxy index seemed to be superior to standard flap monitoring for predicting flap survival.³⁹ In human forehead flaps, HSI mapping traced the expected decrease in ${\mathrm{SO}}_2$ during partial and full excision of the flap.⁴⁰ In diabetes, the blood amount in the foot dermis and ${\mathrm{SO}}_2$ based on multispectral spatial frequency domain imaging could stratify subjects with varying severities of the disease.⁴¹ Furthermore, with the same technique, a low amount of blood in the superficial dermis and a high ${\mathrm{SO}}_2$ indicating poor oxygen transfer to tissue and a heterogenous light scattering were predictive of leg ulcer risk in a diabetic population.⁴²

Other optical techniques for estimating ${\mathrm{SO}}_2$, e.g., near-infrared spectroscopy⁴³ and HSI,^22,27,44,45 often need to be calibrated using standard occlusion provocations or phantom experiments in which the ${\mathrm{SO}}_2$ level is known or measured with a reference technique. This is mainly due to the use of either simplistic inverse algorithms based on modified Beer–Lambert’s law expressions or pure empirical expressions in which only two or a few spectral components are analyzed. These algorithms are also typically sensitive to variations in tissue optical properties, including scattering and melanin content. The method presented in this work addresses these shortcomings through a thorough modeling of both instrumental effects and skin tissue variations.

5.

Conclusion

Our results demonstrate that, with precise modeling of both hardware characteristics and tissue reflectance, ANN can estimate ${\mathrm{SO}}_2$ from MSI data with an accuracy comparable to inverse MC, a method often referred to as the gold standard for analyzing bio-optical signals. The use of ANN, trained on synthetically generated data, is computationally much more efficient than inverse MC and enables real-time ${\mathrm{SO}}_2$ imaging. The ANN approach is, however, more sensitive to imperfections in the modeling of the sensor response and requires noise and other hardware variations to be included when generating synthetic training data.

6.

Appendix: Supplementary Video

Video 1: Example of blood ${\mathrm{SO}}_2$ imaging from the ANN analysis showing an occluded (60–360 s) and reperfused (>360 s) middle finger (MP4, 11.9 MB [URL: https://doi.org/10.1117/1.JBO.29.S3.S33304.s1]).