8.10.3 Expression Modeling for Invariant 3D Face Recognition

Bronstein et al. [15] developed expression-invariant face recognition by modeling facial expressions as surface isometries (i.e. bending but no stretching) and constructing expression-invariant representations of faces using canonical forms. The facial surface is treated as a deformable object in the context of Riemannian geometry. Assuming that the intrinsic geometry of the facial surface is expressioninvariant, an isometry-invariant representation of the facial surface will exhibit the same property.

The Gaussian curvature of a surface is its intrinsic property and remains constant for isometric surfaces. (Clearly, the same is not true for mean curvature.) By isometrically embedding the surface into a low-dimensional space, a computationally efficient invariant representation of the face is constructed. Isometric embedding consists of measuring the geodesic distances between various points on the facial surface followed by Multi-Dimensional Scaling (MDS) to perform the embedding.

Once an invariant representation is obtained, comparing deformable objects, such as faces, becomes a problem of simple rigid surface matching. This, however, comes at the cost of loosing some accuracy because the facial surface is not perfectly isometric. Moreover, isometric modeling is only an approximation and can only model facial expressions that do not change the topology of the face, such as an open mouth.

Given a surface in discrete form, consisting of a finite number of sample points on surface S, the geodesic distances between the points dij = d(xi , xj ) are measured and described by the matrix D. The geodesic distances are measured with O(N ) complexity using a variant of the Fast Marching Method (FMM) [80] which was extended to triangular manifolds in [52]. The FMM variant used was proposed by Spira and Kimmel [83] and has the advantage that it performs computation on a uniform Cartesian grid in the parameterization plane rather than the manifold itself.

Bronstein et al. [15] numerically measured the invariance of the isometric model by placing 133 markers on a face and tracking the change in their geodesic and Euclidean distances due to facial expressions. They concluded that the change in Euclidean distances was two times greater than the change in geodesic distances. Note that the change in geodesic distances was not zero.

The matrix of geodesic distances D itself can not be used as an invariant representation because of the variable sampling rates and the order of points. Thus the Riemannian surface is represented as a subset of some manifold Mm which preserves the intrinsic geometry and removes the extrinsic geometry. This is referred to as isometric embedding. The embedding space is chosen to simplify the process. Bronstein et al. [15] treat isometric embedding as a simple mapping:

between two surfaces such that the geodesic distances between any two points in the original space and the embedded space are equal. In the embedding space, geodesic distances are replaced by Euclidean distances. However, in practice, such an embed-

Fig. 8.15 The canonical representations (second row) are identical even though the original face surface is quite different due to facial expressions (top row). (Image courtesy of [15])

ding does not exist. Therefore, Bronstein et al. [15] try to find an embedding that is near-isometric by minimizing the embedding error given by:

where dij and dij are the distances between points i, j in the embedding and original spaces respectively. X = (x1, x2, . . . , xN ) is an N by m matrix of parametric coordinates in Mm and W = (wij ) is a symmetric matrix of weights determining the relative contribution of the distances between all pairs of points to the total error. The minimization of the above error with respect to X can be performed using gradient descent.

In addition to the limitations arising from the assumptions discussed above, another downside of this approach is that the isometric embedding also attenuates some important discriminating features which are not caused by expressions. For example, the 3D shape of the nose and the eye sockets is somewhat flattened. Figure 8.15 shows sample 3D faces of the same person. Although the facial surface changes significantly in the original space due to different facial expressions, the corresponding canonical representations in the embedded space look similar.

The approach was evaluated on a dataset consisting of 30 subjects with 220 face scans containing varying degrees of facial expression. The gallery consisted of neutral expressions only and the results were compared with rigid face matching [15].