Head Tracking:

IVC Logo

In this work, we propose an algorithm for 3D head tracking that extends the range of head motion allowed in the planar model. Our system uses a texture mapped 3D surface model for the head. During tracking, each input video image is projected into the surface texture map of the model. Model parameters are updated via robust image registration in texture map space. The output of the system is the 3D head parameters and a 2D dynamic texture map image. The dynamic texture image provides a stabilized view of the face that can be used for facial expression recognition, lip reading, and other applications requiring that the position of the head is frontal and almost static.

The system has the advantages of a planar face tracker (reasonable simplicity and robustness to initial positioning) but not the disadvantages (difficulty in tracking large rotations). The main differences are that a.) self occlusion can be managed and b.) better tracking of the face can be achieved through the use of a texture map mosaic acquired via view integration as the head moves.

The following is a brief overview of the head tracking approach. Readers are referred to the technical report. for a detailed description of the approach.

basic idea
Figure 1: Mapping from image plane to texture map.

Our technique is based directly on the incoming image stream; no optical flow estimation is required. The basic idea consists of using a texture mapped surface model to approximate the head, accounting in this way for self-occlusions and to approximate head shape. We then use an image registration technique to fit the model with the incoming data.

To explain how our technique works, we will assume that the head is exactly a cylinder with a 360 degree wide image, or more precisely a movie due to facial expression changes, texture mapped on its surface. Obviously only a 180 degree wide slice of this texture is visible in each frame. If we know the initial position of the cylinder we can use the incoming image to compute the texture map for the currently visible portion, as shown in Fig. 1. The transformation to project part of the incoming frame in the corresponding cylindrical surface depends in fact only on the 3D parameters of the cylinder and on the camera model.

As a new frame is acquired it is possible to find a set of cylinder's parameters such that the texture extracted from the incoming frame best matches the reference texture. In other words, the 3D head parameters are recovered by performing image registration in the model's texture map. Due to the rotations of the head the visible part of the texture can be shifted respect to the reference texture, in the registration procedure we should then consider only the intersection of the two textures.

input input input
output output output
texture map texture map texture map

Figure 2: Example of head tracking and recovered dynamic texture map.

The registration parameters determine the projection of input video onto the surface of the object. Taken as a sequence, the project video images comprise a dynamic texture map , as shown in Fig. 2. This map provides a stabilized view of the face that is independent of the current orientation, position and scale of the surface model.

At this point the tracking capabilities of this system are only slightly better than that of a planar approach, because a cylinder is a better approximation of a face respect to a plane. The key to allowing for large rotation tracking consists of building a mosaicked reference texture over a number of frames, as the head moves. In this way, assuming that there are no huge interframe rotations along the vertical axis, we always have enough information to keep the registration procedure working. The resulting mosaic can also be used as input to face recognition.

For details of the head tracking formulation, readers are referred to the technical report.

Click here to see more head tracking examples.


  1. S. Sclaroff and J. Isidoro. "Active Blobs". In Proc. ICCV 98, January 1998.

  2. M. La Cascia, J. Isidoro, and S. Sclaroff. Head Tracking via Robust Registration in Texture Map Images . In Proc. CVPR 98, June 1998.

[ Head Tracking Home | Contact | IVC Home ]

© 1998 Image and Video Computing Group - Boston University
Last Modified: March 5, 1998