2.1.

Visual Fatigue Description in Stereoscopic

A binocular vision is produced when we use two separate images corresponding to the left and right eyes, although slightly different, merged in viewer’s brain to build a common impression.¹³ Hodges and McAllister¹⁴ describe the method of right and left perspective view in the 3-D display. Based on binocular parallax, the 3-D screen that can be implemented, relies on the format of the image presented and the viewing format. Figure 2 illustrates the watcher experiencing a stereoscopic sensation on images depending on presenting the appropriate view to each eye on a 3-D screen. Also, by improving depth perception, we can feel an added realism for stereoscopy. Although stereoscopic imagery can be presented on 3-D displays, it violates the relationship of natural viewing in the real world. In Fig. 2, the viewer observes a real object or an image on a two-dimensional (2-D) device, the eyes accommodate (focus on) and converge to a specific point. Accommodate distance matches with the convergence distance. Conversely, a viewer obtains a stereoscopic image on 3-D display, the remaining focus point is also on the plane of screen, while the eyes convergances of the image are located at a different distance. Because of the breakdown of the relationship between the accommodation and convergence, a visual discomfort is caused.

Fig. 2

Comparison between stereogram viewing and natural viewing.

For 3-D comfort evaluation, Choi et al.¹⁵ identify some factors to capture the spatiotemporal characteristics of disparity. The prediction of visual comfort is determined by factors fusing. Figure 3 illustrates types of disparity during stereoscopic viewing. Two disparities are indicated on the coordinate plane, positive (uncrossed) and negative (crossed) disparities by blue and red zone.¹³ In Fig. 3, the horizontal gray line position of display represents zero disparity planes. A zero disparity plane is a converged domain of stereoscopic imaging, and also the zero disparity area is commonly referred to as a comfortable zone of stereoscopic imaging.^16,17 Depending on the stereoscopic disparity, different 3-D imaging positions can be implemented, such as in front of or behind the screen. Stereoscopic disparity refers to the difference in image location of one object viewed by the left and right eyes. When a 3-D camera captures a stereoscopic image, each lens separately converges on the main object, and generates stereoscopic disparity. The main object can be seen as a single image, but the background would be seen as double images with disparity.

Fig. 3

Relationship between (a) positive disparity and (b) negative disparity; (c) is natural scene.

In Fig. 3(a), the positive disparity in the stereoscopic image corresponds to the uncrossed line. In Fig. 3(b), the negative disparity corresponds to the crossed line. The negative disparity exhibits crosstalk that occurs between accommodations of each eye. In addition, the negative disparity shows a larger disparity and object size than positive disparity, since the imaging in negative disparity is closer than in positive disparity. This phenomenon is related to the geometry of a binocular viewing. Therefore, negative disparity can incur more visual fatigue than positive disparity.^17,7 Yilmaz and Gudukbay¹⁸ point that the crosstalk (or ghosting effect) is the faded image viewed by the untargeted eye. This effect is undesirable because it may cause visual fatigue and other problems. Gudukbay and Yilmaz¹⁹ indicate that a more comfort stereo view can be achieved in terms of reduced crosstalk (or ghosting effect).

2.2.

Visual Fatigue Measurement Model Description

Body fatigue can be easily tracked from observable physiological features.^20,21 This scheme is considered the relatively objective method for visual measurement. Physiological features may be classified into: The contactless and the contact features. Contactless features contain the EMs, head movement, etc., and these movements can be easily detected from a real-time monitor. Contact features contain the brain activity, heart rate variability, etc., and these movements can be detected by EEG, ECG, and other bio-sensor systems.

The CLPF-based scheme focuses on inferring the fatigue from the contactless features. Ji et al.²² demonstrate that the human in fatigue should exhibit some visual cues in long-time visual experiments. Horng et al.²³ present a fatigue measurement algorithm depend on the eye tracking and dynamic matching. Kim et al.²⁴ construct a neural network-based scheme for fatigue recognition by detecting the movement of the mouth and eyes, respectively.

The CPF-based scheme focuses on inferring the fatigue from the contact features. For example, the EEG can represent abundant information on the human cognitive states, according to the detection in the major EEG bands ($\delta$, $\theta$, $\alpha$, and $\beta$). Lal et al.²⁵ present a fatigue recognition algorithm on different levels of EEG bands. Also, Jung et al.,²⁶ and Wilson and Bracewell²⁷ propose a method to estimate and predict the fatigue level based on the EEG power spectrum estimation and fuzzy neural network model. According to the main electroencephalography (EEG) activities ($\delta$, $\theta$, $\alpha$, and $\beta$) for 52 subjects (36 males and 16 females) during fatigue measurement, Budi et al.²¹ found that $\delta$ and $\theta$ activities is stable over time, but there is a slight decrease for activity of $\alpha$, and a significant decrease for activity of $\beta$. For the other important CPF ECG signal, in Refs. 28 and 29 fatigue recognition refer to heart exhibition on low frequency (LF), very low frequency (VFH), high frequency (HF), and the LF/HF ratio.

Previous physiological feature-based schemes focus only on a single specific aspect. That may lead to inaccurate results because the fatigue is not directly observable, which can only be inferred from the information available. There are a number of reasons for the inaccuracies using the scheme mentioned above: (1) Contextual factor. Fatigue recognition contains much subjectivity that cannot always reflect the real objectivity. (2) Environment factor. For example, when human is present in a not well acquainted environment,³⁰ an inaccurate interpretation of the facial expression (such as eye and mouth movement) would be caused, especially for the introverted persons. Therefore, to fuse as many as possible features from uncertain events is a better way to make an accurate inference.³¹ Further, Picard et al.³² figured out that it was necessary to fuse the contextual and physiological features and the human performance in order to make the fatigue measurement more reliable.

By considering the evidence and beliefs of the contextual information and physiological features from measurement, Ji et al.²² construct a BN-based algorithm to infer and predict the fatigue of human beings, enhancing the reliability of fatigue detection. Yang et al.²⁰ develop a BN-based fatigue recognition model to be used in systems that evolve over time. However, such visual fatigue network in Refs. 20, 22, 33^–36 mostly apply to driving, visual display terminals monitoring, and marine industry. To the best of our knowledge, there is no relating issue on stereoscopic visual fatigue based on probabilistic framework or BN. Eventually, considering the states and beliefs of contextual information and physiological features, a novel probabilistic framework-based (the BN-based) measurement model for stereoscopic visual fatigue is proposed in this article.

2.3.

Bayesian Networks Method Description

Hubbard³⁷ describes uncertainty as the lack of certainty, a state of having limited knowledge where it is difficult to infer precisely the existing state or future outcome. Decision making is generally recognized by engineers as an indispensable part of the whole engineering design process. Just as most fatigue recognition, the stereoscopic visual fatigue measurement is also comprised of a number of uncertainty factors. Because of the fact that uncertainty has a significant impact on judgment, the engineer tries to manage uncertainty via compound methods and intelligent systems. The most reliable tool for modeling uncertainty is the use of probabilities theory.³⁵

One of the most prevalent and effective graphical models to manage uncertainty is the BNs.³⁸ A BN, belief network or directed acyclic graphical model, is a probabilistic graphical model that correlates the conditional dependencies of a number of random variables with the use of a Directed Acyclic Graph (DAG). A DAG is a directed graph with no directed cycles. The formation of a DAG includes vertices and directed edges, each edge connecting one vertex to another so that a cyclic route is impossible to appear. Figure 4 shows an implementation of DAG in our application.

Fig. 4

The detailed BN structure used to measure visual fatigue in stereoscopy.

The basic concept in the Bayesian treatment of certainties in causal networks is conditional probability. Whenever a statement of the probability $P(A)$ of an event $A$ is given, then it is given conditioned by other known factors. Therefore, according to the feature vector mentioned above and conditional probability, the probability of estimated fatigue is obtained through Bayesian theorem in Refs. 20 and 39:

3.1.

Specifying the Nodes of the Discrete Bayesian Networks

As remarked in Fig. 4, there are many contextual and physiological features related to fatigue. Among these features, some of them lead to more contributions to fatigue while others have lesser contributions to the fatigue. For the sake of simplicity but without any loss of generality, we only select those contextual and physiological features that have immediate relations with the fatigue measurement. In particular, the following features are described in step 1. For the contextual, hidden and observable selected in Fig. 4, the fuzzy method is used to determine the discrete values for each variable based on a set of heuristic knowledge rules.⁴⁰

3.1.1.

Stereoscopic contextual features node

Binocular disparity (BD) node. Lambooij et al.⁴¹ noted that the human eye experiences conflict between the accommodation and vergence that mostly affect visual fatigue in stereoscopy. Ohzawa et al.¹³ classified the disparity as positive disparity and negative disparity. Kim and Cho⁷ suggested a simplified relative visual fatigue metric that considers the “accommodation and vergence” factors that can be calculated by the disparities in stereoscopy. We are motivated by Ohzawa et al.¹³ and Kim and Cho.⁷ As exhibited in Fig. 5, several sets of different stereoscopic instances were provided to evaluate visual fatigue. The different sample image in the negative disparity zone and in the positive disparity zone has been shown in experiment for 3-D fatigue measuring.

Fig. 5

(a) Test images to evaluate visual fatigue and (b) the graph of mean opinion score result with various Avg. converged objects disparity and comfort zone in stereoscopy in Ref. 7. Note: The valuation is based on the five grades (1 to 5); 1: very comfortable, 2: comfortable, 3: a little uncomfortable, 4: uncomfortable, 5: very uncomfortable.

Display quality (DQ) node. As Michel et al.¹² described, with 3-DTV and 3-D cinema at the extremes of the screen size spectrum, comfort zone issues for stereoscopy are different when trying to use them to present the same content. Apparently, resolution and luminance are also key elements of display. For example, an unsuitable resolution and luminance also causes a visual discomfort. However, among these features, the screen size has immediate relations with the DQ on issues that are our concern as mentioned in Refs. 12 and 42. Therefore, the display size is taken as a main contextual features corresponding to the DQ nodes.

3.1.3.

Observation state node

EEG node. In the frequency domain, the EEG mainly includes the $\delta$ band (0.5 to 4 Hz) corresponding to the sleep activity, the $\theta$ band (4 to 7 Hz) that is related to drowsiness, the $\alpha$ band (8 to 13 Hz) corresponding to relaxation and creativity, and the $\beta$ band (13 to 25 Hz) that corresponds to activity and alertness. Budi et al.²¹ note that the $\beta$ band has strong relations with visual fatigue. Through the variations in the EEG tracing, the power of $\beta$ frequencies increase as watching duration increases, and it is much stronger in 3-D rather than in 2-D conditions, as shown in Fig. 6(a). Li et al.⁴⁴ identified that the 3-D content affected the power of brain wave in the $\beta$ frequency. The $\beta$ power was stronger at viewing the 3-D contents. Also, subjective results also showed more strong visual fatigue in the 3-D condition than in the 2-D condition. Therefore, we take the waveband magnitude of the EEG spectrum in the $\beta$ band as an observable variables node in BN diagram.

Fig. 6

Physiological response: 3-D and 2-D compared, 3-D viewed first. ($x$-axis is time, $y$-axis is magnitude).

EM node. The EM-based visual fatigue measurement is related to such quantities such as eye gaze, eye blink, and eyelid closure. These manifestations are described in Ref. 45 for the fatigue detection. Zhu and Lan²² pointed out that EM is a reliable and valid determination of fatigue. In Ref. 46 the percentage of eyelid closure over the pupil in a given time (PERCLOS) is indicated. It illustrates that the viewer is possibly in a state of fatigue if the eyes are at least 80% closed during a period of 1 min. Thus, the proportion of the eye-closed time was taken in this article as one of the observable variables corresponding to the nodes of the BN diagram.

3.2.

Determining Discrete Variables in Each Node

The construction of BN has two tasks: one is the determination of nodes; and the other is the determination of its parent discrete variables and their states for each note. In the previous step, the related nodes are determined. While in the following section in step 2, we describe the discrete variables and their states that indicate the likelihood of a particular feature that contributes to the fatigue.

Visual fatigue node: $Z=[Z_1,Z_2]$ in which $Z_1$ and $Z_2$ represent the fatigue and no-fatigue states, respectively.

Contextual features node: $X=[X_1,X_2,X_3]$ represents the nonstereoscopic factor node state, in which $X_1$, $X_2$, and $X_3$ represent the sleep quality, CR and EE, respectively. Here, $X_1=[X_{11},X_{12}]$ in which $X_{11}$ and $X_{12}$ represent the sleep parameters, including the sleep time and sleep satisfaction. $Y=[Y_1,Y_2]$ represents the stereoscopic factor node, in which $Y_1$ and $Y_2$ represent the binocular disparity and DQ, respectively.

Observation features node: $O=[O_1,O_2]$ represents the observation features node, in which $O_1$ represents the CLPF (e.g., EM), and $O_2$ represents the CPF (e.g., EEG).

As remarked in Fig. 4, $z_k$, $x_i^j$, $y_i^j$, and $o_i^j$ denote the specific values taken by $Z=[Z_1,Z_2]$, $X=[X_1,X_2,X_3]$, $Y=[Y_1,Y_2]$, and $O=[O_1,O_2]$, respectively. In Fig. 4 the variables, together with the directed edges, form the DAG. $P(x_i^j)$ represents the probability of the sleep quality node states $\{x_1^1=\text{good},x_1^2=\text{bad}\}$, CR node states $\{x_2^1=\text{active},x_2^2=\text{drowsy}\}$ and EE node states $\{x_3^1=\text{comfortable},x_3^2=\text{uncomfortable}\}$; $P(y_i^j)$ represents the probability of the binocular disparity node states $\{y_1^j=\text{disparity zone}\}$ and DQ node states $\{y_2^1=\text{small},y_2^2=\text{large},y_2^3=\text{ex-large}\}$; $P(o_i^j)$ represents the probability of the contact physiological node states (EEG node) $\{o_2^1=\text{decrease},o_2^2=\text{no-change},o_2^3=\text{increase}\}$ and contactless physiological node states (EM node) $\{o_1^1=\text{large},o_1^2=\text{medium},o_1^3=\text{small}\}$.

3.3.

Calculating Bayesian Networks

Assume that the evidences from the contextual nodes are represented as $e^{X,Y}=\{e_{XY}^{ij}\}$, and the evidences from the observable nodes are represented as $e^O=\{e_O^{ij}\}$, where $e_{XY}^{ij}$ represents the evidence of the $i$’th contextual node with the $j$’th state value ($x_i^j$ and $y_i^j$), and $e_O^{ij}$ represents the evidence of the $i$’th observable node with the $j$’th state value ($o_i^j$). $e=\{e^{XY},e^O\}$ as evidences from the contextual factor and observable feature nodes, respectively. In Eqs. (2) and (3), $P(Z=z_k|e^{XY})$ is the prior probability of visual fatigue $Z$ that was inferred before the parents’ contextual evidence was available. $P(e^O|Z=z_k)$ is the conditional probability of observable evidence $e^O$, if the parent visual fatigue $Z$ turns out to be true.

Then the conditional probability of $Z$ given the occurrence of the $e^{XY}$ node can be written as in Ref. 39

The conditional probability of $e^O$ given the occurrence of node $Z$ can be written as in Ref. 39

According to the BN theorem,⁴ the conditional probability of node $Z$ given the occurrence evidence can be obtained by combining Eqs. (2) and (3); and it can be written as in Ref. 39.