Structured light illumination systems can measure a target’s two-dimensional height profile. Recent advancements in computational power and algorithms have enabled the predesign of optical systems where all experimental variables are determined prior to use in the laboratory. We present a predesign process to align the projector of a custom-structured light illumination system. The predesign method uses the native optical design file to simulate the residual aberrations and can perform a trade space analysis to tailor the alignment variables to desired preferences. The presented method can predict the experimental focus position to 1.6 μm root mean square over a broad range of physical settings. |
1.Introduction1.1.OverviewDetermining the height profile of a target is useful in many applications, such as industrial production where rapid feature recognition is key to high fidelity and low cost. One common noncontact method to produce height profiles uses structured light illumination (SLI),1,2 in which a pattern of light from a projector illuminates the object and camera images the scene. The pattern varies spatially and is further distorted by the sample’s three-dimensional (3D) shape. Image processing techniques then reconstruct the 3D image and produce the desired height maps.3 As with most optical systems, accurate optical alignment of SLI systems is key to the reconstruction resolution and performance. Recent advances in computer power and algorithms have enabled the predesign of optical systems. Côté et al.4 created a deep-learning enabled framework called LensNet that extracts features of successful lens designs and then recombines the features to create new layouts. Ivanov et al.5 published a method to determine the number of optical surfaces necessary to achieve diffraction-limited performance prior to any actual optical design. Oleszko and Gross6 used a numerical model to study freeform optical systems surface-by-surface and correct for aberrations beyond the fourth-order. This paper applies the predesign process to projector alignment in custom SLI systems. The projector is aligned with the quad target method (QTM),7 a simple height reconstruction technique that produces an estimate of the projector’s focal position using only a single camera image. The presented predesign method uses the native optical design file to include the effect of residual aberrations, a key step for accurate simulation of the experimental behavior. The goal here is to quickly move from the design to the lab by enabling a user to completely design the alignment process variables prior to any lab work, even tailoring the responses with a trade space analysis. The simulated alignment process is compared with experimental measurements on a custom SLI system to determine the accuracy of the predesign method. Results suggest that the predesign method can accurately model the experimental behavior. This paper is organized into five sections. The remainder of Sec. 1 briefly describes SLI and the QTM. Then, Sec. 2 displays the test system since the optical design must be well-established prior to utilizing the presented predesign method. Section 3 presents the predesign process including the coordinate system and a simulation flowchart. Section 4 describes the experimental methods and displays and compares the experimental and simulated results. Conclusions are found in Sec. 5. 1.2.Structured Light IlluminationOne common method to perform 3D profilometry is structured light illumination (SLI). A schematic of a general SLI system is shown in Fig. 1.2 Here, the structured light projector images a particular pattern onto the target. Many patterns are found in the literature, such as binary,8 gray-level,9 and colored bar patterns.10 A popular subset of SLI is fringe projection profilometry (FPP) where the projected pattern is sinusoidal fringes.11–13 The rectangular target distorts the color bars in Fig. 1 due to its height above the substrate. A camera then images the entire scene. The combination of the camera image and a processing method enables the reconstruction of the target’s 3D shape.1,2 1.3.Quad Target MethodThis paper utilizes the quad target method (QTM),7 an alignment technique that features a linearized response to focus the projector of an SLI or FPP system. The linearization is achieved by sampling the field at two different locations along the optical axis using the quad target. The quad target is a array of flat surfaces in pairs of quadrants separated by a step height as seen in Fig. 2. Then, focus is efficiently linearized by measuring the difference in the projected fringe frequency contrasts between the two quad target heights. QTM can produce an estimate of the focus position using only a single image, a improvement in speed over the common three-phase reconstruction technique.3 Thus, QTM provides linear feedback with a clear zero-crossing, to minimize any focus ambiguity, and fast positional feedback, to minimize adjustment lag. QTM is governed by two input variables: the frequency of the projected fringes and the height of the quad target step ; it outputs two quantities: the slope of the linearized values and an estimate of the current focus position . A schematic of QTM is shown in Fig. 2. Two field points (red and blue) of sinusoidal fringes at frequency are imaged by the projector optics onto the quad target, a array of flat surfaces physically separated along the optical axis, with two of the four steps shown in profile in Fig. 4. Notice that due to the quad target step height (greatly exaggerated for viewing ease), the blue field is in focus and the red defocused. A camera images the entire scene with the camera’s front working distance labeled . The distance is the projector’s back working distance which can be physically altered to shift the projector’s focus position during alignment. Figure 3 shows an example output of QTM. The raw and fitted fringe contrast values from the front step are in blue with the same curves for the back step are in red. The raw data is fit to a fifth-order polynomial (dotted lines) for noise reduction and robustness with a curve shape. Note that in this paper the fitted data curves for the front and back steps are called the and curves, respectively. Next, the linearized response (orange asterisk) is calculated from the and curves and also fit to fifth-order polynomial (orange dotted line) for the same reasons. Then, the nonlinear coefficients of the fit are set to zero to ensure that is linear near the calibrated focus position at . Finally, the black dotted line shows the current focus position of , i.e., where by definition of QTM.7 As stated above, the raw , and curves in Fig. 3 are all individually fit to fifth-order polynomials. For the given input variables, the curves in Fig. 3 are well-approximated by a second-order polynomial leading to the potential for errors caused by over-fitting. However, Ref. 7 showed that fifth-order fits do not lead to over-fitting for a wide range of QTM input variables. Additionally, proper choice of the projected light level and the large averaging described in Sec. 3.1 should ensure no over-fitting of the curves. 2.Test System2.1.LayoutThis paper begins with the test system, since the design must be established before the alignment method can be predesigned. The custom SLI sensor14 shown in Fig. 4 has two cameras and four projectors, meaning that there are eight total SLI channels. Throughout this paper, however, only a single projector/camera combination was examined for brevity. A listing of the optical attributes of the sensor is found in Table 1. Note that the optical design is outside the scope of this paper. Table 1List of optical attributes for the custom SLI sensor.
In Table 1, two fields of view (FOVs) are listed: 3D and raw. The 3D FOV is the lateral region where the sensor can measure and generate 3D height maps, whereas the raw FOV is the full FOV of the camera. The raw FOV is slightly larger than the alignment FOV to ensure good alignment across the 3D FOV and to reduce edge effects during height reconstruction. 2.2.Through Focus PerformanceAs will be described in Sec. 4, an important aspect in predicting the alignment performance is the through-focus response of the test system. Figure 5 shows the through-focus modulation transfer function (MTF) contrasts for three spatial frequencies using the native optical design. These specific frequencies will be tested in Sec. 4. Notice that the highest spatial frequency of 13.7 cy/mm in Fig. 5(a) has a much lower depth of field than the lowest frequency of 3.4 cy/mm in (c), and that the contrast dynamic range for 13.7 cy/mm is much larger than 3.4 cy/mm (where “dynamic range” is the difference between the maximum and minimum values). Thus, the tradeoff with is higher spatial frequencies resulting in more sensitivity but narrower Z range for adjustment. 3.Simulation Method3.1.Coordinate SystemThe alignment predesign is performed via simulation. The coordinate system for the simulation method is shown in Fig. 6. The alignment FOV (red box) is slightly larger than the system FOV (green box), as described in Sec. 2.1. The black boxes show the experimental regions of interest (ROIs), i.e., the ROIs utilized during the projector alignment. The experimental ROIs are set as large as possible to reduce noise while avoiding the edges of the quad target step. To match the experimental ROIs as much as possible, the simulated field points are placed at the center of the four ROIs, thereby assuming that the aberrations across the ROI are roughly balanced at the center. 3.2.Simulation FlowchartThe simulation method uses the native optical design file. With this approach, the residual aberrations are accurately modeled, and their effects are included in the predicted response. Therefore, simulating with the native file is a key step for accurate simulation of the experimental behavior. The simulation raytracing was performed in Zemax OpticStudio;15 using the ZOS-API, a MATLAB16 script managed the data flow and graphed the results. Figure 7 shows a flowchart of the simulation code. The QTM variables of and are set to the desired values. Then, the target plane is shifted by to simulate the front and back surfaces of the quad target, respectively. Next, the contrasts are determined for the four field points shown in Fig. 6 using the operand MTFT since the triangulation axis and tangential axis are parallel. The and contrasts are determined by multiplying the contrasts from the projector and camera,17 specifically where the and are the contrasts of the projector and camera, respectively. Finally, the linearized value is determined via QTM.Distortion is of particular concern to SLI systems because it causes the projected fringe frequency to change across the FOV and, in turn, alters the measured fringe contrast. Since the native optical design file is used in the simulation method, the distortion’s effect is already included in the prediction; therefore, there is no need to correct the camera’s distortion for the presented simulation method. If additional distortion is found experimentally, the unmodeled distortion can be mitigated by decreasing the ROIs in Fig. 6 to avoid the FOV edges where distortion is at its largest magnitude, or by increasing the width of the bandpass filter used to determine the contrast to ensure that the altered frequency is measured. 3.3.Simulation OutputsWith this method, the test system’s response can be simulated. Figure 8 displays the , and curves through focus for and . This graph shows the position along the axis; the normalized contrast on the left-hand axis; and the linearized value on the right-hand axis. The main two outputs of QTM are garnered from Fig. 8: the slope of the linearized values and the current projector focus position . The slope affects the resolution and “feel” of the adjustment for the user. The current focus position is estimated using the linear fit of . By definition of QTM, the current focus position is where , so the focus position is estimated as where and are the coefficients for linear slope and bias, respectively.3.4.Trade Space AnalysisWe can probe the effect of and with a large-scale trade space analysis.18 Here, both variables are altered and the predesign method determines and . This process has a unique benefit for the user: it allows the user to tailor the alignment method for their preferences. For example, if a user desires maximum adjustment sensitivity, the user might employ the highest projected fringe frequency to maximize . 4.Experimental and Simulation Results4.1.Experimental ProcessTo test the accuracy of the predesign method presented in Sec. 3, experimental measurements of and were taken using the test system in Sec. 2. As shown in Table 2, seven different combinations of step heights and spatial frequencies were examined to sample the two variables across a wide range. Specifically, covers , , and of the camera’s depth of field and covers , , of the projector’s maximum spatial frequency (referring to Table 1). Notice that combinations of low and low and mid are not included since Ref. 7 showed that these variable combinations do not produce accurate estimates of focus position over ranges of . Table 2Table of hstep and ξproject combinations tested both experimentally and in simulation.
Figure 9 shows two of the three tested quad targets resting below the test sensor. The targets are displaced in to change the distance in Fig. 2. For all experimental and simulated results in Sec. 7, the range of values was with a step size. The stage is a Newport GTS30V and is controlled via a MATLAB script. Finally, the test sensor is rigidly mounted to a granite frame to reduce vibrations, as seen in Fig. 4. 4.2.Single ComparisonAs an initial comparison between the experiment and the predesign method, Fig. 10 shows the experimental curves from Run #3 with the simulated curves from Fig. 8. The error bars in Fig. 10 are from four experimental trials. Overall, the predesign method seems to predict the experimental curves well, as evidenced by the good visual agreement and high values in Table 3. Figure 10 also shows the most straightforward method to demonstrate the simulated and experimental agreement since it only uses one set of input variables. Table 3Table R2 values from a linear regression of the experimental and simulated curves for Run #3.
4.3.Trade Space AnalysisAs described in Sec. 3.3, the predesign method can perform a trade space analysis of the variable space. The analysis studied and over ranges, with starting from the maximum fringe frequency and decreasing , and starting from the camera’s depth of field and increasing. Then, to further examine the predesign method’s accuracy, we compared the experimental and simulated values for and . 4.3.1.Linearized slopeOne of the two outputs of QTM is the slope of the curve . Figure 11(a) shows the simulated slope and (b) the localized gradient of the surface in (a). The mean angle of the gradient is 59.0 deg meaning that has a slightly stronger effect on than . Thus, if a user desires the alignment to be more sensitive, i.e., a higher , increasing the fringe frequency or step height will have roughly the same effect. The seven combinations in Table 2 were tested experimentally using the test system in Sec. 2. The results are displayed graphically in Fig. 12 with summary statistics in Table 4. The graph exhibits very good linear regression (the dotted line) both visually and with . Furthermore, the root mean square (RMS) difference for all runs is and the regressed slope are very close to ideal at 1.04. These results suggest that the predesign method can accurately predict the linearized slope and seems insensitive to the and combination. Table 4Table of linearized slope numerical results from trade space analysis.
Table 5Table of best focus numerical results from trade space analysis.
4.3.2.Focus locationThe other output from QTM is the estimate of the current focus location . Figure 13 shows the output of the tradespace analysis for and Fig. 16(b) the localized gradient of the surface. Unlike linearized slope, the location changes very little for the variable ranges except for the lower right-hand corner, i.e., low and high . For the gradient, very strongly affects the focus position with a mean gradient angle of . Furthermore, most of the vectors are nearly 90 deg, meaning that is essentially the only effect on . This is likely caused by the projector’s MTF having different responses for different , as seen in Fig. 5. The dynamic range of the values in Fig. 13(a) is . While only 10.4% of the camera’s depth of field, that range may be too large for SLI systems that require micron-focus precision. Looking back to the through-focus curves in Fig. 5, we find that focus positions for and 3.4 cy/mm are not equal and in fact are separated by . Thus, the range of in Fig. 13(a) test system is due to the test system’s frequency-dependent focus position and not an error within the predesign method. As in Sec. 4.3.1, was measured for the seven combinations using the same experimental procedure. was determined with Eq. (2) with the results shown graphically in Fig. 14 and numerically in Table 5. Figure 14 displays a noticeable correlation between experimental and simulated focus locations, with the regressed slope at 1.05. However, the correlation is not as strong as with likely due to experimental noise. The predicted accuracy of is very high with an RMS difference of over the seven combinations, thereby suggesting that the predesign method can accurately estimate the current focus location. 4.4.Alignment Design RecommendationThe presented results suggest that the alignment process can be simulated. Thus, we can now predesign the QTM variables using the results in Figs. 12 and 14. For the tested SLI system, maximum adjustment sensitivity is desired more than the usable range due to the relatively small lateral resolutions of thus, and . 5.ConclusionsThe predesign of a linearized projector alignment method has been realized using simulation. The presented predesign method uses the native optical design file to determine the projector alignment variables so that a user may quickly move from design to a functioning alignment system. The method was compared with experimental results over nine different combinations of projected fringe frequencies and quad target step heights. The simulation method predicted both experimental outputs of the QTM accurately with the linearized slope RMS difference of and the current focus position to RMS. AcknowledgmentsThe author would like to thank J. Konicek, B. Christensen, S. Schmidt, and J. Sapp for fabricating the quad targets. The author would like to acknowledge E. Rudd, E. Jungwirth, and T. Skunes for helpful comments on this manuscript. The authors have no conflicts of interest to report. This paper is an expansion of Ref. 19. ReferencesJ. Wang and Y. Liang,
“Generation and detection of structured light: a review,”
Front. Phys., 9 688284 https://doi.org/10.3389/fphy.2021.688284
(2021).
Google Scholar
J. Geng,
“Structured-light 3D surface imaging: a tutorial,”
Adv. Opt. Photonics, 3 128
–160 https://doi.org/10.1364/AOP.3.000128 AOPAC7 1943-8206
(2011).
Google Scholar
J. Salvi, S. Fernandez and T. Pribanic,
“A state of the art in structured light patterns for surface profilometry,”
Pattern Recognit., 43
(8), 2666
–2680 https://doi.org/10.1016/j.patcog.2010.03.004
(2010).
Google Scholar
G. Côté, J. Lalonde and S. Thibault,
“On the use of deep learning for lens design,”
Proc. SPIE, 12078 120781A https://doi.org/10.1117/12.2603656 PSISDG 0277-786X
(2021).
Google Scholar
S. Ivanov, K. Choi and A. Morozov,
“Pre-design optical lens complexity estimation method,”
Proc. SPIE, 11871 1187119 https://doi.org/10.1117/12.2602137 PSISDG 0277-786X
(2021).
Google Scholar
M. Oleszko and H. Gross,
“Zernike surface contributions as an assisting tool for designing freeform optical systems,”
Proc. SPIE, 10690 106901C https://doi.org/10.1117/12.2312529 PSISDG 0277-786X
(2018).
Google Scholar
M. E. L. Jungwirth, E. P. Rudd and E. Yudovina,
“Linearized projector alignment system for structured light illumination systems using the quad target method,”
Opt. Eng.,
(2022).
Google Scholar
I. Ishii et al.,
“High-speed 3D image acquisition using coded structured light projection,”
in IEEE/RSJ Int. Conf. Intell. Rob. and Syst.,
925
–930
(2007). https://doi.org/10.1109/IROS.2007.4399180 Google Scholar
S. Inokuchi, K. Sato and F. Matsuda,
“Range-imaging for 3-D object recognition,,”
in Int. Conf. Pattern Recognit.,
806
–808
(1984). Google Scholar
K. L. Boyer and A. C. Kak,
“Color-encoded structured light for rapid active ranging,”
IEEE Trans. Pattern Anal. Mach. Intell., 9
(1), 14
–28 https://doi.org/10.1109/TPAMI.1987.4767869 ITPIDJ 0162-8828
(1987).
Google Scholar
S. Zhang and P. Huang,
“Novel method for structured light system calibration,”
Opt. Eng., 45 083601 https://doi.org/10.1117/1.2336196
(2006).
Google Scholar
M. Vo et al.,
“Hyper-accurate flexible calibration technique for fringe-projection-based three-dimensional imaging,”
Opt. Express, 20 16926
–16941 https://doi.org/10.1364/OE.20.016926 OPEXFF 1094-4087
(2012).
Google Scholar
P. Lu et al.,
“Accurate and robust calibration method based on pattern geometric constraints for fringe projection profilometry,”
Appl. Opt., 56 784
–794 https://doi.org/10.1364/AO.56.000784 APOPAI 0003-6935
(2017).
Google Scholar
Multi-Reflection SuppressionTM (MRSTM) Sensor Technology, CyberOptics Corp., Minneapolis, Minnesota www.cyberoptics.com Google Scholar
OpticStudio, Version 22.1.2, Zemax LLC, Kirkland, Washington
(2022). Google Scholar
MATLAB, Version 9.12.0.1927505 (R2022a) Update 1, The MathWorks Inc., Natick, Massachusetts
(2022). Google Scholar
T. B. Andersen and Z. A. Granger,
“Camera System MTF: combining optic with detector,”
Proc. SPIE, 10375 103750C https://doi.org/10.1117/12.2272832 PSISDG 0277-786X
(2017).
Google Scholar
M. E. L. Jungwirth, D. V. Wick and E. L. Dereniak,
“Theory and tradespace analysis of a reflective axial adaptive optical zoom system,”
Opt. Eng., 51
(8), 083001 https://doi.org/10.1117/1.OE.51.8.083001
(2012).
Google Scholar
M. E. L. Jungwirth, S. Schmidt and J. Sapp,
“Quad target method: optical simulation to model the step height and projected fringe frequency,”
Proc. SPIE, 11816 1181602 https://doi.org/10.1117/12.2596333 PSISDG 0277-786X
(2021).
Google Scholar
BiographyMatthew E. L. Jungwirth is a senior optical scientist at CyberOptics in Minnesota. He earned his PhD from the University of Arizona in 2012 studying adaptive optical zoom using carbon fiber mirrors. He has 17 patents and 11 publications covering a diverse array of topics such as barcode sensing, quantum information processing, and star tracking technology. He is an SPIE senior member and associate editor of both Optical Engineering and the Spotlight book series. |