2.1.

Basic Principle of Laser Light Section Method

The laser light section method (LLSM), shown in Fig. 2, is an extensively used alternative to stereo vision, in which one of the binocular cameras is generally replaced by a light source. One or multiple structured light patterns are projected onto a test object and captured by a camera. The intersection of the camera ray with the corresponding light plane can yield 3D coordinates of point $P$ only if the imaging geometry is calibrated. Development based on this method benefits the miniaturization of the projection hardware, and the prototype can thus be designed as a compact unit for applications in confined spaces.

Fig. 2

Principle of the LLSM. A light plane from the projector illuminates the object and is captured by a camera. The 3D coordinates of point $P$ relative to the camera system can be computed after calibration.

Sensitivity to depth is generally determined by the geometry of triangulation in the LLSM, especially by the length of the baseline.^12,13 Our previous study showed that a shorter baseline indicates a larger systematic error associated with the detection of the distance.¹⁴

Additional beam scanning is needed with one-or two-dimensional (2D) relative motions between the detector and test object to measure the 3D shape of the object. However, if test objects have deep holes or are located in confined spaces, the relative motion cannot be easily measured. Consequently, developing a unique 3D measurement/imaging method is necessary for acquiring the 3D shape of an object with an ultracompact system configuration. Notably, a compact configuration will lead to a short baseline, and the change in baseline will increase systematic error. Hence, the designing of a compact optical configuration with a preferable precision is one of the highlights of this study. In addition, the measurement of the 3D shape of an object alone is insufficient in some surgical inspections. The focus should also be on the needs of surgeons, such as shape rotation, zooming ability, and even relative position relationships between the endoscope’s distal tip and other intraoperative tools.

2.2.

Endoscope Design

2.2.1.

Imaging fiber-based optical configuration

Figure 3 shows the optical configuration of the endoscopic 3D scanner using the LLSM method. The optical configuration mainly consists of three parts: the light source projection, coupling optics, and scanner’s distal tip. For the first part, we adopted a helium–neon laser projector supplied by Daheng Optics (DH-HN, Beijing, China) as the laser scanning light source. The projected laser has a wavelength of 632.8 nm, since the knee-bone has the highest reflection at this wavelength. The diameter of the laser line must remain as small as possible to achieve a high-resolving power. Hence, the laser line is first focused using lens L₁, and then projected onto the proximal end surface of the imaging fiber, wherein the coupling optics consist of two lenses. As a result, the laser line is delivered via the flexible imaging fiber and transmitted to the reference plane through the probe imaging lens mounted on the distal-end surface of the optical fiber. In addition, the optical fiber was manufactured by Asahi Kasei Corporation (SPN, Tokyo, Japan) and comprised 13,000 pixel elements with a distal-end diameter of 1.0 mm. Baseline $b$ indicates the distance between the distal-end surface of the optical fiber and optical axis $z$ of the camera. In this case, $b$ was an extremely short distance of 1.4 mm.

Fig. 3

Optical configuration of the developed endoscopic 3D scanner based on LLSM (${\mathrm{L}}_{1-3}$, lens; PIL, probe imaging lens; $b$, baseline; RP, reference plane, with the middle one as the focal plane).

2.2.2.

Ultracompact camera

The applicability of the proposed 3D endoscopic scanner in a confined space cannot be ensured using a commercially available, normal-sized camera that does not have a compact design or high precision. For example, in Ref. 15, the image sequence captured by a normal camera demonstrated an extremely high noise level and periodic pulsation. If the system operates in an underwater environment, the camera would not be able to acquire any images effectively. Therefore, a customized camera (ultracompact camera) with a diameter of 1.8 mm and its control circuit were designed for the proposed fiber-based endoscopic imaging, as shown in Fig. 4. To avoid causing illumination artifacts, four white micro-LEDs were mounted radially around the front of the camera. This camera is ideal for medical applications owing to its wide field-of-view with ultrashort focus distance. Specifications of the camera are listed in Table 1.

Fig. 4

Photograph of the (a) distal tip of the camera and the LEDs and (b) the customized control circuit (size: $7\mathrm{cm}\times 2.5\mathrm{cm}\times 0.5\mathrm{cm}$).

Table 1

Specifications of the ultracompact camera used.

Parameter	Value
Cube chip sensor	OVM6946
Resolution	$400\times 400\text{pixels}$
Optical size	$1/18\text{in}$
Minimum focal distance	2 mm
Field-of-view	$120\mathrm{deg}(\text{horizontal})\times 60\mathrm{deg}(\text{vertical})$
Video frame rate	160 K pixel @ 30 frames per second
Pixel size	$1.75\mu \mathrm{m}\times 1.75\mu \mathrm{m}$
Output interface	USB 3.0
Illumination	Four LEDs

According to the optical configuration shown in Fig. 3, the camera coordinate system and the image plane we defined are both shown in Fig. 5. A point $P$ is captured by camera and is projected onto the projection center of the camera with a corresponding point ($x^{\prime },y^{\prime }$) on the image plane. The relationship of the 3D coordinates of point $P$ ($x_c,y_c,z_c$) in the camera coordinate system can be denoted as follows based on the laser triangulation of the pinhole camera model:

Eq. (1)

$$\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right]=z_c{\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]}^{-1}\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right],$$

where ($c_x,c_y$) is the principal point, and $f_x$ and $f_y$ are the focal lengths in the $x$ and $y$ directions, respectively, which can be obtained following the calibration of the camera.

Fig. 5

Definition of the camera coordinate system.

As the imaging fiber is located on the top of the camera (Fig. 3), the imaged laser line only appears on the top half of the image plane. Based on the camera coordinates, if light spots were captured successfully, then $0\ll y^{\prime }\ll c_y$ and $z_c\gg 0$. In addition, the imaged light spots located on the top half of the image plane are negative in the $y_c$, and those on the left half of the image plane are negative whereas those on the right half are positive in the $x_c$, as shown in Fig. 5. Thus, the depth distance can be expressed as follows:

Eq. (2)

$$z_c=\frac{b{f}_y}{|y^{\prime }-c_y|}=\frac{b{f}_y}{c_y-y^{\prime }}.$$

where $b$ is the baseline length with a value of 1.4 mm.

2.3.

Electromagnetic-Based Stereo Imaging Method

Similar to many monocular structured light systems, the scanning results of the current frame (e.g., frame $n$) will be replaced immediately by those of frame $n+1$ with respect to the movement of the endoscope because the camera is not equipped with a functionality to save previous data. Furthermore, a common problem in monocular endoscopic imaging is the initialization, because the depth information of a scene cannot be recovered from only one single image frame.¹⁶ Hence, a unique method for 3D imaging is essential in our design to save the complete scanned data and display the shape of the objects accurately.

We propose a real-time stereo imaging method for 3D shape measurements using a state-of-the-art electromagnetic-based tracking system (LIBERTY, Polhemus, United States). The system consists of a transmitter and microsensor. The transmitter produces an electromagnetic field that acts as an accurate reference for position and orientation measurements of the microsensor. The microsensor (with a diameter of 1.6 mm) can offer high-fidelity continuous tracking with six degrees-of-freedom and a maximum update rate of 240 Hz. Furthermore, the microsensor is assembled with a camera and imaging fiber that allows the distal tip of the endoscope to be small with a diameter of 3.4 mm. All three parts were fixed in a simple axially aligned configuration comprising the distal tip.

The difference between our proposed approach and the feature-based tracking method presented in a previous 3D imaging study for minimally invasive surgery^15,17 is shown in Fig. 6, demonstrating the use of different coordinate systems during object imaging. The endoscope is located in the camera coordinate system and aimed at the virtual object (the bone) to be displayed. Figure 6(a) shows the coordinate system of the feature-based tracking method. Once the feature is detected and tracked, the object is imaged in the feature coordinate system. Assuming $P_f$ as a 3D point ($x_f,y_f,z_f$) on the bone surface, the 3D point can be transformed into a 2D point ($x^{\prime },y^{\prime }$) in the endoscopic camera’s view using the following equation:

Fig. 6

The comparison of the (a) marker-less tracking method and (b) our proposed EBSIM tracking method.

In our proposed EBSIM, as can be seen in Fig. 6(b), we add a microsensor local space $S$ as an agent, which serves as the intermediary and is incrementally built from the point cloud sensed in the environment, enabling the achievement of significant robustness in data collection. Let ${\mathbf{P}}_{\mathbf{s}}={[x_s,y_s,z_s]}^T$ describe the same point, $P$, with respect to the microsensor coordinate system as follows:

Euler angles, namely, azimuth, elevation, and roll are represented by $\alpha$, $\beta$, and $\gamma$, respectively. These angles represent an azimuthal primary sequence of frame rotations (counterclockwise rotation) that define the current orientation of the microsensor corresponding to the zero-orientation state of the transmitter. Thus, the rotation matrix of the current orientation of the sensor is expressed as follows:

Equations (4) and (5) show that the 3D coordinates of point $P$ with respect to the transmitter coordinate system are denoted as

Real-time imaging reconstruction can be performed on a frame-by-frame basis on the 3D coordinates of a specimen’s surface scanned by the camera using Eq. (6) and the OpenGL technique. In addition, the reconstructed 3D shapes, including shape rotation, zooming, and offline observations based on the unified transmitter coordinate system, are available for online processing (Sec. 3.3).

2.4.

Characteristic Curve of the Proposed Endoscopic Scanner

2.4.1.

System constants

The characteristic curve of the proposed endoscopic scanner is formulated in this section. The curve describes the relationship between the depth distance and location of the imaged light spot of the camera detector. Using Eq. (2), the corresponding point, $y^{\prime }$, can be expressed as follows:

Eq. (7)

$$y^{\prime }=-b{f}_y\frac{1}{z_c}+c_y=A\frac{1}{z_c}+B,$$

where $A=-b{f}_y$ and $B=c_y$ are the system constants of the endoscopic scanner. The values of the system constants can be determined based on the calibration results shown in Sec. 3.1.

Equation (7) indicates that point $y^{\prime }$ is not uniform but has an inverse linear relationship with respect to depth distance $z_c$. This property hinders the operation of a 3D endoscopic scanner. Furthermore, any change in the corresponding point $y^{\prime }$ will result in a systematic error; the larger the change, the larger the error. Therefore, the corresponding point is an important parameter when designing a 3D endoscopic scanner.

To reveal the relationship between the point $y^{\prime }$ and the depth distance $z_c$, a spot was projected onto a white flat board vertically by the scanner when located on a moving stage. The details of the components setting and operations were same as that for the flat-plane evaluation experiment described in Sec. 3.2. By doing so, we can obtain the relationship between a corresponding point and the illumination distance. Figure 7 shows the ideal characteristic curve of the proposed endoscopic scanner. However, the actual working characteristics of the endoscopic scanner must be considered thoroughly. A series of illuminated spots, $z_c$, and their corresponding points, $y^{\prime }$, were measured, as shown in Fig. 7, for comparison with the results obtained by Eq. (7). The results from the measured positions are consistent with the ideal characteristic curve. In addition, the measuring range of the endoscopic scanner in the $z_c$-coordinate range of 2 to 21 mm is imaged on the camera’s detector length at a $y$-coordinate, of $\sim 100\mathrm{mm}$ in this case. The measurement ranges for the $x_c$ and $y_c$ coordinates correspond to the fixed field-of-views of 120 deg and 60 deg, respectively, regardless of the measurement distance.

Fig. 7

Relationship between the coordinate of image spot $y^{\prime }$ and the position of light spot $z_c$ on the specimen of the prototype.

2.4.2.

Error propagation

The displacement of the illuminated spot $z_c$ by amount $d{z}_c$ leads to the displacement of the corresponding image point by $d{y}^{\prime }$, which is dependent on the illuminated distance as a consequence of the nonlinear characteristic depicted by the curve obtained using Eq. (7). We considered the derivation of function $y^{\prime }=y^{\prime }(z_c)$ from Eq. (7) with respect to $z_c:d{y}^{\prime }/d{z}_c$.

According to the law of error propagation, the standard deviation of image spot coordinates ${\sigma }_{y\prime }$ on the image plane is defined as follows:

Eq. (8)

$${\sigma }_{y^{\prime }}^2=a^2{\sigma }_{z_c}^2(a=\frac{d{y}^{\prime }}{d{z}_c}),$$

where ${\sigma }_{z_c}$ is the standard deviation of illuminated spot $z_c$, and $a$ is the gradient of the relationship between the illuminated spot $z_c$ and the $y^{\prime }$-coordinate of the image spot. It can be rewritten as

Eq. (9)

$${\sigma }_{y^{\prime }}=a{\sigma }_{z_c}.$$

Figure 8 shows the relationship between the gradient and standard deviation of the image spot coordinate, ${\sigma }_{y^{\prime }}$, based on the illuminated spot $z_c$ coordinate. Gradient $a_1$ is considerably smaller than $a_2$ in this figure. In this case, standard deviation ${\sigma }_{y^{\prime }1}$ becomes smaller than the standard deviation ${\sigma }_{y^{\prime }2}$ when standard deviations of $z_{c1}$ and $z_{c2}$ are constant at ${\sigma }_{z_c}$. Therefore, the gradient is a vital factor in considering the measurement accuracy and range when utilizing a 3D endoscopic scanner. Moreover, the scanning of specimens with the maximum working distance of 12 mm can achieve high precision.

Fig. 8

Relationship between the gradient and standard deviation of image spot coordinates ${\sigma }_{{\mathrm{y}}^{\prime }}$ based on illuminated distance ${\mathrm{z}}_c$.

3.1.

Experimental Configuration

The 3D shape measurement was performed using the hardware setup shown in Fig. 9. The control circuit mainly consists of a decoding circuit for the OVM 6946 chip sensor and a driver for the light-emitting diode (LED) illumination, with adjustable brightness. The microsensor is connected to the electronic unit by a cable for data exchange. The transmitter is placed at an optimal distance of 1 m from the microsensor during the data-collection process. Figure 9 also shows the front view of the endoscope tip. The camera, fiber, and microsensor were fixed in a tube of diameter 3.4 mm. The $e=1.7\mathrm{mm}$ is a user-defined distance between the camera and microsensor, which can yield the translation vector ${\mathbf{T}}_{\mathbf{cs}}$ given in Eq. (4).

Fig. 9

Hardware setup of the measurement system.

The pinhole camera was calibrated prior to the verification of the proposed endoscope scanner using Zhang’s method as used in Ref. 18. The employed chessboard has a pattern accuracy of 0.01 mm. The comparison of the chessboard capture before and after calibration is shown in Fig. 10. The radial lens distortion shown in Fig. 10(a) is large owing to the large field-of-view of the camera. Figure 10(b) shows the precise correction of the distortion after camera calibration. The intrinsic camera matrix obtained by calibration was as follows:

Fig. 10

Comparison of image capture (a) before and (b) after camera calibration. The size of the chessboard is $18\mathrm{mm}\times 13.5\mathrm{mm}$ with a square size of 1.5 mm.

3.2.

Flat-Plane Evaluation

3D scanners are commonly evaluated to develop a set of procedures that use artifacts with a common geometry, such as planes, spheres, and cones.¹⁹ Liquid crystal displays were assumed as flat plane specimens to evaluate the precision of the endoscopic scanner.²⁰ First, the test plane was positioned in front of the scanner. Second, the scanner was located on a handwheel translational stage (GCM-83 Daheng Optics, Beijing, China) and was moved in a controlled fashion along its $z$-axis. In addition, the $z$-axis was parallel to the normal direction of the reference plate. Thus, the position of the scanner was changed precisely with a positioning resolution of 0.01 mm.

Reference planes were scanned from 2 to 12 mm with intervals of 2 mm. This range was chosen based on the actual distance within which surgeons usually conduct observations in actual medical cases.¹⁰ The reference planes were placed perpendicular to the optical axis of the endoscope to obtain the optimal focus of the laser. Table 2 lists the average depth distributions and measurement errors between the plane positions and measured results. These results indicate that the measurement error of the flat plane was $<0.18\mathrm{mm}$, with the standard deviation of $<0.018\mathrm{mm}$, which shows a slight improvement compared with the values obtained for a distance of 8.00 mm.

Table 2

Measured results of reference planes.

The captured image of the laser beam and the extracted center lines are shown in Fig. 11. As the laser beam was very short, we indicate the local amplification effect for a better observation. Figure 12 shows the depth distribution results of the flat-plane with a scanning distance of 8 mm.

Fig. 11

Captured image sample of (a) laser beam, (b) area detection of the laser beam, and (c) extracted center line of the laser.

Fig. 12

Measurement results of flat-plane.

3.3.

Cylindrical Surface Evaluation

A cylindrical wood specimen with a diameter of 40.05 mm, shown in Fig. 13(a), was used to test the performance of the endoscopic scanner system. The shape was machined via CNC milling. A quarter of the cylindrical surface was radially and manually scanned at a distance of $<12\mathrm{mm}$ with the endoscopic sensor. Figure 13(b) clearly depicts the 3D data of the obtained surface. The original 3D raw point measurement data contained 49,778 points. However, the standard evaluation criteria or standard specimens for 3D point cloud data of the endoscope are lacking. We then employed a geometric algorithm to fit a set of 3D points with the tested cylinder.²¹

Fig. 13

Measured results of the cylindrical surface: (a) photograph of the measured wood and (b) point-by-point 3D representation of the scanned surface.

Figure 14 shows the representation of a cylinder. The cylinder was specified by an axis containing a point $C$ and having unit-length direction $\mathbf{W}$. Moreover, $r$ was the radius of the cylinder. Two more unit-length vectors $\mathbf{U}$ and $\mathbf{V}$ were defined such that $\{\mathbf{U},\mathbf{V},\mathbf{W}\}$ was a right-handed orthonormal set and each vector was mutually perpendicular. Thus, any 3D point $P$ can be written uniquely as

Fig. 14

Representation of a cylinder.

Let ${\{{\mathbf{P}}_i\}}_{i=1}^n$ be the cylindrical point set. An error function for a cylinder is expressed as $E={\sum }_{i=1}^n{(r_i^2-r^2)}^2$. Setting the partial derivative of the error function with respect to the squared radius to zero, we then obtained the constraint

The experimental results indicate that the diameter of the obtained cylindrical surface is 39.81 mm. The deviation from the cylindrical surface was 0.24 mm with a standard deviation of 0.031 mm. An error of approximately two times the error of the flat plane was observed in the measurement of the cylindrical surface mainly because the manual measurement of the flat plane was static, while that of the cylindrical test was based on dynamic scanning. Although unrelated to the original intention (manual scanning) of the endoscopic development, the precision of the measurement can be improved slightly if the experiment is designed to use a microuniversal stage to scan shapes with cylindrical surfaces.

Keeping the endoscope perpendicular to the surface is difficult because of the variable scanning distance. The poor focus of the projected structured light results in a weaker detection and data losses. The nonuniform movement during manual scanning is another factor that must be considered. Figure 15 shows the position tracking with six degrees-of-freedom and orientation of the endoscope during the measurement of the cylindrical surface. The tracking curves for the $x$- and $y$-positions are clearly not as smooth as those of the 3D orientation. Hence, an excessively fast scan will cause data loss. Accordingly, an extremely slow scan could result in repeated scans at the same location and duplicated data. Our previous endoscope characterized image data through nonperiodic fluctuations. These fluctuations during the detection of light intensity are caused by the exposure or frame drops of the camera. These situations often occur in web cameras. Therefore, this feature was eliminated by locking the exposure parameters, which were set in the customized control circuit.

Fig. 15

Six degree-of-freedom tracking of the endoscope.

The endoscopic scanner software was developed on the Windows platform based on EBSIM and then combined with a rendering algorithm.²² The software interface is shown in Fig. 16. The left side of the image shows the 3D raw point measurement data for the surface. The two other forms can provide users with shape rotation, tilting, and zooming of the scanned surface. Figure 17 presents the rendered surface from different viewpoints. The recovered surface quality is highly promising.

Fig. 16

User interface of the 3D point cloud measurement of surface and observation software.

Fig. 17

Observation effect of the rendered surface at different viewing angles: (a) front and (b) left views.

3.4.

Knee-Joint Test

An ex vivo knee-joint of a chicken bone sample with an approximate size of $1.5\mathrm{mm}\times 1\mathrm{mm}$ was used to demonstrate the functionality of the endoscopic system and obtain the 3D model (see Fig. 18). To simulate the intraoperative inspections of the knee-joint, this measurement test was conducted in an underwater environment. The measurement was completed in $<2\mathrm{s}$ and the collected data consisted of 103,672 points. The point density is significantly higher (more than two times) than that of the cylindrical surface owing to the better reflection of light compared with that of the cylindrical surface. The experimental results showed that the endoscopic scanner works even on biological surfaces that cannot be easily scanned owing to volume scattering and highlights. The data quality demonstrates potential because the curvature of the complete joint can be recovered. However, in a microscopic environment, measuring the surface of tissue with high reflectivity is inevitable. Reflective highlights from the water cause some interferential data, as shown by the black arrow in Fig. 19. We attribute the possible reasons to the following two points. For one thing, the material of the knee surface was smooth and characterized by reflection of light. For another, measuring a surface under water with a short scan distance made the camera more sensitive to the light intensity. There are mainly two types of techniques that may optimize the measurement results for such problems. One technique is based on changing the illustration light source intensity or based on camera exposure techniques.^23–25 The other is to employ a filter in front of the camera tip. The former method is more complicated for operation because it requires to fine-tuning the projection and exposure parameters several times to find a best effect before measurement. However, the exposure settings of the camera cannot be adjusted in some cases. Moreover, low intensity of the light may probably lead to a worse detection of the structured light. In contrast, the latter method is easier to implement if economic cost is disregarded.

Fig. 18

Ex vivo knee-joint chicken bone sample.

Fig. 19

3D point cloud measurement data for knee-joint surface of a chicken.

To quantitatively evaluate the surface congruency of the measured knee-joint, an assessment method that is based on the measured 3D point sets must be designed for free-form surfaces. This will be considered in our future work.