2.1.

Optical Flow Formulations

Our approach to optical flow estimation is based on applying a differential constancy constraint equation¹⁵ to the images

The optic flow components $v_x,v_y,v_z$ and the brightness variation factor $\alpha$ are now to be estimated altogether. To this end, we apply a discrete wavelet transform (DWT) of the image: $I\to {\omega }_{s,j}=I\ast {\psi }_{j,s}$ , where $j=1,\dots ,n_j$ runs over the number of scales used in the DWT, and $s=0,\dots ,7$ labels the polarizations of the wavelet filters ${\psi }_{j,s}$ (there are eight such filters in three dimensions). Equation 2 then yields a set of constraints (one constraint for each wavelet filter),

In practice, local irregularities sometimes occur even in areas where the aperture problem is expected not to affect results. To reduce such irregularities, the optical flow map was low-pass filtered with a 3-D Gaussian kernel with a standard deviation of $8\times 8\times 4\text{pixels}$ . For our images, this kernel size turned out to be a good compromise, allowing irregularities to be substantially reduced without spoiling the optical flow accuracy by oversmoothing.

All computations were performed using $64\text{-bit}$ Matlab (the Mathworks, Natick, Massachusetts), running on a Linux PC with $24\mathrm{GB}$ of random access memory.

2.2.

Animal Preparation and Visualization

The preparation used in these experiments has previously been described in detail.^{18, 11, 19} In brief, temporal bones from young anesthetized guinea pigs were excised, using procedures approved by the local ethics committee (permit no. N319/06). The bulla was opened, and the preparation attached to a holder in a chamber containing tissue culture medium (Minimum Essential Medium with Hank’s salts, Gibco, Paisley, Scotland). A small opening was made in the basal turn of the cochlea [see schematic drawing in Fig. 1 ], and a thin piece of plastic tubing inserted in scala tympani (ST) to allow perfusion of oxygenated medium and fluorescent dyes, which were used to label the inner ear structures. The sensory hair cells and auditory nerve fibers were labeled by the styryl dye RH795. Supporting cells were stained by the cytoplasmic dye calcein/acetoxymethyl ester (both from Molecular Probes, Leiden, the Netherlands). An additional opening in the apical part of the cochlea allowed visualization of the organ of Corti while providing exit for the medium.

Fig. 1

(a) Drawing showing a cross section of the cochlea and the arrangement for perfusion, with a tube inserted in ST of the basal turn and the reservoir above the fluid level of the chamber. By moving the reservoir, the hydrostatic pressure inside the fluid-filled compartments is shifted; an increased pressure in ST moves the organ of Corti in the direction of scala vestibuli (upward in this image). (b) 3-D maximum brightness projection of a stack of images from the apical turn of a guinea pig cochlea. The three orthogonal axes of the standard coordinate system are superimposed on the image. The longitudinal axis is aligned with the first row of outer hair cells, pointing toward the base of the cochlea. The radial axis points from the center of the cochlea in the direction of the outer hair cells, and the transversal vector is direct to the radial-longitudinal plane. Scale bar, $10\mu \mathrm{m}$ .

The preparation was visualized with a water-immersion objective (Achroplan $40\times$ , numerical aperture 0.8, Zeiss, Jena, Germany) using a laser-scanning confocal microscope (LSM 510, Zeiss) equipped with a $15\text{-}\mathrm{mW}$ krypton/argon laser and a $5\text{-}\mathrm{mW}$ helium/neon laser. To avoid photobleaching and consequent cellular damage, all stacks were acquired at the minimum laser power compatible with an acceptable signal-to-noise ratio in the images. In most experiments, this resulted in a laser setting of $\sim 0.25\mathrm{\%}$ of the maximum power for the $488\text{-}\mathrm{nm}$ laser line and $\sim 10\mathrm{\%}$ for the $543\text{-}\mathrm{nm}$ line.

2.3.

Organ of Corti Motion Evoked by Pressure Changes

ST pressure was changed by moving the perfusion reservoir above and below the fluid level of the preparation in a sequence forming a cycle, typically 0, $+10$ , 0, $-10$ , $0\mathrm{cm}$ , where $0\mathrm{cm}$ corresponded to the surface of the liquid surrounding the preparation. Each position was maintained for a few seconds to allow the hearing organ to reach a new equilibrium position, and a stack of confocal images acquired. The typical pixel format of the stacks was $512\times 512\times 32$ , requiring about $40\mathrm{s}$ to acquire at a pixel dwell time of $6.4\mu \mathrm{s}$ . The spacing between the sections was $0.5–1\mu \mathrm{m}$ .

The pressure changes in the ST are linearly related to the changes in height of the reservoir. They remain within the physiological range and may be assumed a fraction of $1\mathrm{Pa}$ or less.²⁰ These changes induce reproducible, micron-sized movements of the cochlear partition.

In these experiments, the orientation of the cochlea varied and to compare results across experiments, it is necessary to use a standardized coordinate system. The reference frame that we used is an orthogonal coordinate system whose three axes at any particular point along the cochlea are defined as follows. The $x$ -axis, or longitudinal axis, is parallel to the coil of the cochlea and points toward its base; the $y$ -axis, or radial axis, is taken parallel to the plane of the reticular lamina and oriented from the inner hair cells to the outer hair cells; the $z$ -axis, or transverse axis, is taken orthogonal to $x$ and $y$ . Figure 1 shows the coordinate system superimposed on a maximum brightness projection of a stack of images acquired at different focal planes within the organ of Corti.

3.1.

Validation of the Optical Flow Algorithm

The optical flow algorithm was evaluated on two different synthetic image sequences, one showing a sine pattern and the other a more complex template image showing cochlear structures. Both images had a maximal amplitude of 1 and minimum amplitude of zero. A uniform translation was applied to each pattern, using bicubic interpolation. The exact motion is therefore known, and a direct measure of the error in the optical flow estimation is possible. Because noise is unavoidably present in real image sequences, Gaussian or Poisson-distributed noise was added to each frame following interpolation. No systematic difference between the two types of noise was noted. The Gaussian noise had a mean of zero and standard deviation $\left(\sigma \right)$ ranging between 0.01 and 0.04.

Results are illustrated in Figs. 2, 2, 2 for the sine sequence. The standard deviation of the Gaussian noise in Fig. 2 was 2.5%, representing an image with noise levels typical of those found experimentally. In the absence of added noise, the magnitude error, which is defined as the difference in Euclidean vector magnitude of the estimated and true motion vectors, was found to be $<10\mathrm{\%}$ for all displacements of $<8$ voxels [Fig. 2 solid line]. A systematic downward translation of the entire curve was observed when the noise level increased. At intermediate noise levels, this downward translation resulted in curves with $<10\mathrm{\%}$ magnitude error throughout the range tested.

Fig. 2

Motion tracking on two different sequences of 3-D test images. Motion was generated through bicubic interpolations (a)–(c). An artificial sine pattern was used. No brightness variation is present, but noise added as indicated by the legend inside panels (b) and (c) (d)–(f). Identical image sequence as in (a)–(c) but the overall brightness was reduced by 2% for each motion step. Following brightness reduction, noise was added as in (a)–(c) (g)–(i). An image stack from the hearing organ was subjected to the same manipulations as the sine image in panel (d)–(f). The magnitude error is defined as the root-mean-square distance between the observed and the real motion vectors, relative to the size of the real motion. The angular error is obtained by taking the arccos of the dot product of the real and estimated 3-D motion vectors.

It is also important to determine the direction of motion. The angular error [Fig. 2] is a measure of the difference in orientation of the estimated and true motion vectors in the plane that they span. The angular error was found to be $<10\mathrm{deg}$ at all tested noise levels and displacements. For this artificial image, the direction of motion was therefore more precisely estimated than its size.

The above computations were performed on an image sequence that lacked variations in brightness over time. To simulate an actual experiment with bleaching due to laser exposure, additional tests were performed on sequences where the image amplitude was reduced by 2% for each motion step. After interpolation and brightness reduction, Gaussian random noise was added as described above. The results are shown in Figs. 2, 2, 2. The left half of Fig. 2 is identical to the one shown in Fig. 2. To give a measure of the magnitude of the brightness variations, the right half of Fig. 2 shows the final image in the series, where brightness was reduced to 56% of the original. Apparently, the algorithm can handle such brightness variation. The main effect is a reduction in the slope of the magnitude error curves [Fig. 2], which results in larger magnitude errors at high noise levels. However, angular errors were almost unchanged [Fig. 2]. All the curves remain $<10\mathrm{deg}$ , except for large displacements at the highest noise level. To estimate the improvement from using the brightness-compensated algorithm, we also performed computations where brightness compensation was removed from the algorithm. When using sequences with the above reduction in brightness, we found a substantially reduced performance for small motions ( $<3$ voxels): The magnitude errors increased and so did angular errors. However, for larger displacements, the results were similar to those obtained with the brightness-compensated algorithm. The performance did not depend on the pattern of brightness reductions: A linear decrease in brightness over time gave the same results as a monoexponential one.

The moving sine pattern is an unrealistic test case because it contains only a single spatial frequency. Sharp boundaries are absent, making this sequence difficult for any gradient-based method. A more realistic test is achieved by using real images, which contain larger spatial variations in brightness.

Figure 2 shows a group of hair cells in situ. Important cellular structures such as the stereocilia bundles at the top of the outer hair cells are clearly visible. Noise levels, brightness variations and imposed motions were identical to those used in Fig. 2, 2, 2. From Fig. 2, it is evident that small displacements are detected with high fidelity when noise levels are low $(\sigma <0.02)$ , but there is an overestimation of motion for the range between 3 and 5 voxels. At higher noise levels, good motion detection is seen at $<5$ voxels, but results deteriorate for displacements larger than this. Thus, in terms of the magnitude error, some noise seems to produce better results than no noise. The reason for this unusual, but quite desirable behavior is not currently known. However, the angular error behaves differently. Near-perfect estimation of vector directions was achieved for artificial motions that lacked noise. Results get systematically worse with increasing noise, but the error remains at $<10\mathrm{deg}$ at all noise levels if the motion is $<4$ voxels.

The factor $\alpha$ in Eq. 2 provides an estimate of the brightness changes in the image sequence. To evaluate the accuracy of this estimate, we first assessed its variability in image sequences that lacked variations in brightness. In this case, the algorithm returned a faithful estimate, provided that motion was $<5$ voxels [Fig. 3 ]. The most precise estimates were obtained when a moderate amount of noise was present. The “no-noise” condition as well as the extreme case of noise with a standard deviation of 0.05 both caused performance deterioration. The brightness estimates therefore appear to behave similarly to the motion estimates in Fig. 2. When actual brightness variations were present, a similar pattern was seen [Fig. 3]. The estimated brightness tracked the real brightness change, and the best estimates were obtained at intermediate noise levels.

Fig. 3

(a) Pixel brightness estimation in the absence of a true change in brightness. The outer hair cell sequence was used. The real change in brightness is given by the thick line. Noise levels were identical to Fig. 2. (b) Same computation as in (a), but this time, a 2% reduction in brightness was present for each motion step.

3.2.

Validation of the Experimental Setup

To assess the stability of our experimental setup, we estimated the random motion of the table and scanning mirrors of the confocal microscope by imaging a test sample repeatedly under identical conditions. The sample was a slide containing roots from convallaria majalis, which have a stable structure with bright fluorescence. A series of 11 three-dimensional image stacks were acquired, at two different magnifications, with a $60\text{-}\mathrm{s}$ pause between each acquisition. Note that averaging was not used, and that the pixel dwell time was kept brief, resulting in images with substantial noise [Fig. 4 ]. The values for the stage motion given below therefore represent worst-case estimates because the noise in the images will lead to increased errors when calculating motion.

Fig. 4

(a) Sample image showing the fluorescent roots of convallaria majalis. Note the relatively high noise level of the image. Scale bar, $20\mu \mathrm{m}$ . (b) Random motion of the sample shown in (a) on the microscope stage. The graph shows the 3-D optical flow trajectory of the stage (black solid trace) averaged over 10 image points; the broken lines are projected trajectories on $xy$ , $yz$ , and $xz$ planes. Note the random character of this trajectory, reminiscent of Brownian motion. The ring marks the starting point of the motion trajectory.

The root-mean-square displacement per step, ${\theta }_1$ , estimated from $np$ selected trajectories on the image, was computed as

The displacement of the scanned volume was estimated with the optical flow algorithm. As seen in Fig. 4, the motion of the microscope stage from one stack to the next appeared to be random, with a root-mean-square displacement per step ${\theta }_1=0.19\text{to}0.23\mu \mathrm{m}$ and a cumulative displacement after 10 steps of ${\theta }_{10}\approx 0.5\mu \mathrm{m}$ (pixel size $0.9\mu \mathrm{m}$ ). Note that motion along the $Z$ -axis, the position of which changes during acquisition of a focus series, was most pronounced. This probably relates to slight imperfections in the focus motor controlling the position of the microscope’s table. We may define a quantity, $n_{\mathrm{eff}}={({\theta }_n∕{\theta }_1)}^2$ , which is interpreted as an effective number of steps, which should be approximately equal to $n$ if the stage was undergoing unconstrained Brownian motion. In Table 1 , it is seen that $n_{\mathrm{eff}}$ is substantially smaller than $n$ , which indicates that the stage and scanning system is in fact physically constrained. Obviously, this is a very desirable feature. During $10\min$ , the system does not wander around its mean position by more than a fraction of $1\mu \mathrm{m}$ , a prerequisite for its use for motion measurements. Note that this constitutes an upper bound on stage motion, as this value will be influenced by any error present in the optical flow calculation in addition to the actual motion of the stage. We conclude that errors caused by random shifts within the microscope were small and affected the optical flow estimates made in the cochlear preparation by less than a few percent.

Table 1

Estimates of the step size along the trajectory shown in Fig. 4. np is the number of image points over which averages are taken; θ1 is the root-mean-square displacement of the stage for a duration of 60s ; θn is the cumulative rms displacement after a duration of n×60s , here for n=10 ; neff=(θn∕θ1)2 is to be interpreted as an effective number of steps, which should be approximately equal to n if the stage was undergoing unconstrained Brownian motion. The small neff indicates that the stage is physically constrained and does not wander around its mean position by more than a fraction of a micron.

3.3.

Optical Flow Estimate of 3-D Organ of Corti Motion

The main motivation for the development of our optical flow algorithm was to analyze the 3-D motion patterns of the complex cellular structures of the hearing organ in response to different kinds of mechanical stimulation, including sound. As a first step, we analyzed the 3-D displacement evoked by quasi-static ST pressure changes. Such pressure changes were used in several previous studies;^{20, 18} this type of stimulation leads to a consistent motion pattern within the organ of Corti, the magnitude of which depends on the quality of the seal between the cochlea and the perfusion tubing, the size of the opening at the apex, and the ST impedance.

Seven different experiments showing pressure change–induced movements of outer hair cells in the apical part of the cochlea were analyzed using identical parameters in the optical flow algorithm. An example showing data from two preparations is given in Fig. 5 . As seen in Figs. 5 and 5, the orientation of these two preparations with respect to the optical axis of the microscope differed substantially. The optical section in Fig. 5 is well aligned with the long axis of the cylindrical outer hair cells, whereas the preparation in Fig. 5 was nearly perpendicular to this, with the short axis of the hair cells nearly parallel to the image plane. The optical flow vectors were therefore converted to the standard coordinate system shown in Fig. 1 where the transverse axis is perpendicular to the reticular lamina, the radial axis points away from the center of the cochlea, and the longitudinal axis is directed along the cochlear spiral, toward its base.

Fig. 5

Pressure-evoked motions of the organ of Corti in situ. (a) and (b) show outer hair cells from two different preparations, the longitudinal and radial motions of which are displayed in (c) and (d), respectively. The preparation shown in (a) was subjected to positive scala tympani pressure changes while negative pressures were applied to the preparation shown in (b). Scale bars, $10\mu \mathrm{m}$ .

Positive ST pressures were used in Fig. 5. The resulting motions of the apex of a first row outer hair cell are plotted with dashed lines in Fig. 5 and 5. The transverse displacement, given on the $Y$ -axis, was largest, with amplitude of $1.5\mu \mathrm{m}$ . There was also a smaller motion component directed to the apex of the cochlea, along the cochlear spiral. This longitudinal component reached an amplitude of $1\mu \mathrm{m}$ . As seen in Fig. 5, the smallest motion was directed to the center of the cochlea. In the preparation shown in Fig. 5, negative ST pressures were used. Despite the very different orientation of this preparation, the main axis of the motion vectors is quite similar to the ones described above, and the transverse component remains the largest one. This gives additional confidence in the optical flow algorithm. Evidently, the motion of the organ of Corti evoked by these pressure changes is complex, with significant components in all three directions.

Confocal microscopy and optical flow makes it possible to assess the deformation of cells inside the hearing organ. Image data were used to generate the 3-D wire-frame models shown in Fig. 6 . Figure 6 shows an inner hair cell with its characteristic pear-shaped cell body, and its apical surface perpendicular to the transversal axis. The original wire-frame model of the cell (red) was deformed according to the computed optical flow map. This generated a second wire frame, shown in gray. Following a pressure change, motion components in all three directions were seen, but the largest component was in the radial direction, and the cell therefore appears to be translated toward the right of the image. Figure 6 shows data from an outer hair cell in the same preparation as Fig. 6. The two cells were close to each other, but not at exactly the same longitudinal position. Note the different scales used in Figs. 6 and 6. At the outer hair cell, a small motion component in the transversal direction was present, but the largest motion was radial. This radial motion is much larger than that found at the inner hair cells. The radial motion was larger near the bottom of the cell, which therefore shows a swinging motion, but little apparent deformation. By comparing the data in Figs. 5 and 6, it is apparent that motions at the surface of the hearing organ, displayed in Fig. 5, are different from those found inside (Fig. 6). In particular, the cell bodies of outer hair cells make sideways motions that are larger than those seen at the surface. These sideways motions were bigger than the surface motion in six out of seven cases.

Fig. 6

3-D reconstructions of (a) an inner hair cell and (b) an outer hair cell. The original positions of the cells are given by the red wire frames, which is then deformed according to the computed optical flow map, to generate the gray wire frame that represents the final position of the cell. Scale bars denote a $10\text{-}\mu \mathrm{m}$ distance along each of the three principal axes, transversal (T), radial (R), and longitudinal (L). (Color online only.)

4.1.

Hearing Organ Motion

Optical flow algorithms have previously been used in several studies in the context of hearing research.^{9, 10, 16, 22, 23} Their use arises naturally from the fact that the hearing organ is a complex structure, the function of which depends on interactions between different cell types. Probing such interactions requires a system capable of at least 2D measurements.

The quasi-static pressure changes investigated in the current study represent a first step toward investigating sound-evoked cellular interactions in the hearing organ. We show here that the motion evoked by such pressure changes is complex, with components in all three directions. At the surface of the hearing organ, the reticular lamina, the transversal motion component was the largest one, followed by the longitudinal one. The longitudinal component was directed toward the apex of the cochlea. This may be a consequence of the anatomical arrangement of cells within the tissue: The Deiters cells are supporting cells with long processes extending in the apical direction from the base of the outer hair cells. This results in longitudinal mechanical coupling that may promote motion directed at the apex, in the forward direction of the traveling wave.

Interestingly, the cell bodies of outer hair cells showed larger motion than those seen at the surface. A similar behavior was observed in a previous study using 2-D optical flow, where motion was provoked by pressure gradients much larger than those utilized here.¹⁸ This behavior also occurs during sound stimulation,¹⁰ and during electrical stimulation of isolated cochleae, where movements inside the hearing organ were substantially larger than those seen at the surface.²⁴ Collectively, these results suggest that radial movements of outer hair cell bodies contribute substantially to the internal mechanics of the organ of Corti.