2.1.

Optical Configuration of the Hybrid LFM

Figure 1 shows the optical configuration of the hybrid LFM. The light source is a 488-nm continuous-wave laser (MLD, Cobolt). The laser first passes through the beam expander and neutral density filter and enters the polygonal mirror scanner (DT-72-290-025, Lincoln Laser), which scans the laser beam horizontally to form a light sheet. Next, the light sheet is relayed to a galvanometric scanner (6220HB, Cambridge Technology) via a 4-f system, i.e., L1 and L2, and scanned vertically over a range of $\pm 40\mu \mathrm{m}$. Then, the laser is coupled to the illumination objective (Plan $10\times /0.30\mathrm{W}$, Nikon), which images the scanning light sheet to the focal region to form a uniformly illuminated imaging volume of $250\times 250\times 80{\mu \mathrm{m}}^3$. For detection, the emissions are first collected by the detection objective (DO) (APO LWD $40\times /0.80\mathrm{W}$, Nikon) and coupled to two imaging arms via a tube lens (L5). The emissions are equally separated by a beam splitter (50:50, Chroma) to a scientific complementary metal-oxide-semiconductor (sCMOS) camera, i.e., sCMOS 1 (Zyla 5.5, Andor), which generates high-resolution 2D images; and a light-field camera, which consists of a microlens array (MLA, APO-Q-P150-F3.5, OKO Tech), macrolens (180-mm F3.5, Canon), and sCMOS 2 (C13440, Hamamatsu). Specifically, the MLA is located at the focal plane of L5, and the macrolens maps the MLA focal plane to sCMOS 2. (It is worthy to note that the two sCMOS cameras have the same pixel sizes of $6.5\mu \mathrm{m}$.) Lastly, the collected 2D and light-field images are combined to form high-resolution light-field images via a custom-developed program.

Fig. 1

Optical configuration of the hybrid LFM.

Figure 2 shows the flowchart of the hybrid algorithm. $I_{\mathrm{LF}}$ ($N_x*N_y*N_u*N_v$) and $I_{\text{Re}}$ ($m{N}_x*m{N}_y$) represents low-resolution light-field images from sCMOS 2 and high-resolution 2D images from sCMOS 1, respectively, where ($x,y$) and ($u,v$) represent the position and angular coordinates; and $m$ is an interpolation constant. $I_{\text{Re}}$ and $I_{\mathrm{LF}}$ are simultaneously obtained by scanning the light sheet vertically over a range of $\pm 40\mu \mathrm{m}$ via galvo within a single exposure time of sCMOS cameras, i.e., 10 ms. To begin,$I_{\mathrm{LF}}$ is resized to $I_{\mathrm{LF}\_\mathrm{High}}$ by a factor of $m$ in the spatial domain via the bicubic interpolation, followed by a 4D fast Fourier transform (FFT) to obtain $G$, i.e., 4D Fourier information. It is worthy to note that $F^n$ and $F^{-n}$ represent the $n$-dimension FFT and inverse FFT (IFFT), respectively. Next, multidepth high-resolution images $I_{\mathrm{Re}\_\mathrm{High}}$ ($n*m{N}_x*m{N}_y$) ($n=\text{number of layers}$) are calculated from $I_{\mathrm{LF}}$ via deconvolution, where the image at the focal plane ($z=0$) is replaced by the high-resolution image $I_{\text{Re}}$. It is worthy to note that based on Fourier slicing theorem FFT of $I_{\mathrm{Re}\_\mathrm{High}}$ represents slices along different directions in the Fourier domain of the 4D light-field data, which is used as constraints for slicing in the 4D Fourier domain $G$. Lastly, a 4D IFFT is performed to reconstruct the 4D Fourier signals $G^{\prime }$ in the spatial domain. By setting the negative and complex numbers to zero, high-resolution light field images, i.e., $I_{\mathrm{LF}\_\mathrm{new}}$ ($m{N}_x*m{N}_y*N_u*N_v$), are obtained. Table 1 presents the pseudocodes of the hybrid algorithm.

Fig. 2

Flowchart of the hybrid algorithm.

Table 1

Pseudocode of hybrid LFM algorithm.

2.2.

Experimental Results

This section presents the experimental results of the proposed hybrid LFM. First, we discuss the resolution enhancement over the standard LFM. For standard LFM, the resolution can be estimated by the size of a microlens ($d_{\text{pitch}}$) and the magnification ($M$) of the DO, i.e., resolution $R_{\text{lateral}}=d_{\text{pitch}}/M=150\mu \mathrm{m}/40=3.75\mu \mathrm{m}$ at the focal plane ($z=0$). It is worthy to note that the lateral resolution decreases when the imaging plane moves away from the focal plane. For the hybrid algorithm, a large $m$ value can directly improve the lateral resolution of the reconstructed images, especially at the focal plane. As $m$ continues to increase, the resolution enhancement effect gradually diminishes; in the meantime, the required computational power increases at a rate of $m^2$. To select the optimal value of $m$, simulated experiments have been devised and conducted in MATLAB based on wave optics and scalar Debye theory.^7,16 In the simulation, we calculate the fluorescence imaging results from sCMOS 1 and sCMOS 2, where the emissions are from an infinitely small spot at the focal plane. Next, the two simulated images are imported to the hybrid algorithm. Specifically, 11 2D images from $z=-40$ to $40\mu \mathrm{m}$ with $8\mu \mathrm{m}$ intervals are generated via deconvolution, where the image at the focal plane is replaced by the 2D image from sCMOS 1. The 11 high-resolution images $I_{\mathrm{Re}\_\mathrm{High}}$ are used as constraints in the Fourier slicing theorem to reconstruct a high-resolution 4D light field image. As shown in Fig. 3, the lateral resolution of the reconstructed images is plotted as a function of the interpolation parameter $m$ and axial positions; and as a reference, the resolution of the standard LFM and diffraction limited are also plotted together in Fig. 3. From the results, one may observe the lateral resolution has been improved over the entire scanning range and the effect becomes more prominent with increasing distances away from the focal plane, i.e., the resolution of the hybrid LFM remains relatively constant over the range in comparison with standard LFM. The resolution in the $z$ axis fluctuates as the hybrid algorithm uses high-resolution deconvoluted images at the selected depths (i.e., $z=\pm 40$, $\pm 32$, $\pm 24$, $\pm 16$, and $\pm 8\mu \mathrm{m}$) as constraints to reconstruct the entire 3D volume based on phase retrieval. As such, better resolution is observed at these positions. From the results, we can confirm the increasing $m$ values will enhance the resolution with a diminishing effect for $m$ larger than 5. Considering the tradeoffs between resolution and computational cost, we select $m=3$ in the hybrid algorithm for imaging experiments in the later sections; this is close to the resolution enhancement limit of the deconvolution algorithm for standard LFM.⁷

Fig. 3

Lateral resolution of the hybrid LFM under various $m$ values along the $z$ axis.

In terms of axial resolution, the hybrid LFM should have comparable resolution with standard LFM. To estimate the axial resolution, we first apply the Sparrow criterion,¹⁷ where the Sparrow limit can be described as the closest distance on the microlens imaging plane that two focal points can be resolved ($R_{\text{obj}}$):

Given $d_{\text{pitch}}=150\mu \mathrm{m}$, $M=40$, and $\mathrm{NA}=0.8$, from Eqs. (1) and (2), we can find $R_{\text{obj}}=12.5\mu \mathrm{m}$ and $N=12$. From a previous study,⁵ the axial resolution ($R_{\text{axial}}$) and maximum axial imaging range ($z_{\text{range}}$) of LFM can be calculated via Eqs. (3) and (4), respectively,

To verify the predicted resolution and range, we performed imaging experiments on $1\text{-}\mu \mathrm{m}$ fluorescent beads (F8819, Thermo Fisher Scientific) at different depths to characterize the lateral and axial point spread functions (PSF) throughout the entire imaging volume. The recorded raw images are processed by the hybrid LFM algorithm. To begin, we first identify the optimal value of $m$ by setting $m$ to different values and recording the corresponding images at $z=32\mu \mathrm{m}$, i.e., an out-of-focus position. The results are shown in Figs. 4(a)–4(c), where $m$ is set to 1, 3, and 5, respectively. From the results, one may observe that the lateral resolution is improved substantially when $m$ is equal or $>3$. When $m=5$, the enhancement becomes visually indistinguishable, which is consistent with our prediction, as shown in Fig. 3.

Fig. 4

Reconstructed lateral PSF at $z=32\mu \mathrm{m}$ for different $m$ values.

We record the lateral and axial PSF images over the entire work volume, followed by image reconstruction ($m=3$) via the hybrid algorithm. Figures 5(a) and 5(b) show the reconstructed lateral and axial PSF, i.e., 5.25 and $15.69\mu \mathrm{m}$, respectively, of the hybrid LFM at $z=-32\mu \mathrm{m}$, i.e., worst-case scenario at the far end of the imaging field. In comparison, Figs. 5(c) and 5(d) show the lateral and axial PSF, i.e., 9.07 and $17.49\mu \mathrm{m}$, respectively, of the standard LFM at $z=-32\mu \mathrm{m}$. Figure 5(e) plots the reconstructed lateral and axial PSF over the entire volume for both the hybrid and standard LFM. From the results, we can conclude the hybrid FLM has effectively improved the lateral resolution by a factor of $\sim 2.5$ over the entire imaging field. The lateral and axial PSF measured at the focal plane ($z=0$) are 1.85 and $9.36\mu \mathrm{m}$, respectively. We also find the hybrid LFM achieves better axial resolution than the standard LFM in the vicinity of the focal region, i.e., $z=\pm 15\mu \mathrm{m}$.

Fig. 5

3D PSF of the hybrid and standard LFM.

In this section, we perform 3D imaging experiments on biological specimens, i.e., fern spores (B17124 R8, Walter Products Inc.), to verify the performance of our hybrid LFM. Figures 6(a)–6(e) show reconstructed images from the hybrid LFM at five different depths, i.e., $-30$, $-5$, 0, 5, and $30\mu \mathrm{m}$, with a field of view of $250\times 250{\mu \mathrm{m}}^2$. (It is worthy to note that $z=0\mu \mathrm{m}$ corresponds to the DO focal plane). In comparison, Figs. 6(f)–6(j) show reconstructed images from the deconvolution algorithm; and Figs. 6(k)–6(o) show images obtained from a standard LFM at the same imaging field. From the results, one may find the hybrid system yields images of better resolution and contrast throughout the imaging field in comparison to the standard LFM. Comparing with the deconvolution method, we find that the hybrid algorithm shows improved resolution around the focal plane, i.e., $z=\pm 5\mu \mathrm{m}$, without any artifact; and comparable resolution at regions away from the focal plane ($z=\pm 30\mu \mathrm{m}$). In terms of computation time, the hybrid algorithm can improve the speed by a factor of 3 to 4 due to the use of sparse deconvolution images, e.g., 10 deconvoluted images in our hybrid imaging process in Fig. 6. For example, when processing a light-field image with a size of $1892\times 1892\text{pixels}$, the deconvolution algorithm uses 56 min to reconstruct a 3D image stack from $-40$ to $40\mu \mathrm{m}$ with a step of $2\mu \mathrm{m}$; in contrast, the hybrid algorithm uses 16 min to reconstruct a 3D image stack over the same imaging field. These calculations are performed on a workstation equipped with Intel(R) Core (TM) i9-7920X CPU at 2.90 GHz, 64 G RAM, and Nvidia Quadro P4000 graphic cards. To compare quantitatively, Fig. 6(p) plots the cross-sectional intensity profiles along the red, blue, and green dashed lines in Figs. 6(e), 6(j), and 6(o), respectively, where the intensity curves are normalized over each image, i.e., Figs. 6(e), 6(j), and 6(o), for fair comparisons. From the results, the calculated signal-to-background ratio (SBR) in Figs. 6(e), 6(j), and 6(o) are 1.55, 1.66, and 1.23 with a mean background signal of 0.32, 0.30, and 0.43, respectively. The imaging results affirm the performance improvement of the hybrid LFM over standard LFM and the deconvolution algorithm (in terms of speed and the resolution at the NIP). It is worthwhile to note that the imaging resolution of the hybrid LFM can be further increased using more than one light-sheet images, where each additional light-sheet image can improve the resolution over a local volume at the expense of increased system complexity and cost.

Fig. 6

3D image of fern spores reconstructed via the hybrid algorithm, standard LFM, and deconvolution algorithm. (a)–(e) Reconstructed cross-sectional images from the hybrid algorithm at $z=-30$, $-5$, 0, 5, and $30\mu \mathrm{m}$, respectively; (f)–(j) reconstructed images from the deconvolution algorithm at $z=-30$, $-5$, 0, 5, and $30\mu \mathrm{m}$, respectively; (k)–(o) reconstructed images from the standard LFM at $z=-30$, $-5$, 0, 5, and $30\mu \mathrm{m}$, respectively; and (p) normalized intensity profiles along the red, blue, and green dashed lines in (e), (j), and (o), respectively. $\text{Scale bar}=50\mu \mathrm{m}$. (See Videos 1 and 2 for demonstration.) (Video 1 [URL: https://doi.org/10.1117/1.JBO.25.10.106502.1] MP4, 303 KB and Video 2 [URL: https://doi.org/10.1117/1.JBO.25.10.106502.2] MP4, 176 KB).