CS 180 Project 3: Face Morphing

Introduction

In this project, I morph my face onto other people, work with a face dataset to compute the mean face of a population, and extrapolate from a population mean to caricaturize myself. Instead of simply averaging the pixels of face images, one can be more intelligent, by providing correspondence points between images and shifting while cross-dissolving. This allows us to connect the eyes, mouth, chin, ears, etc. of each face to the other face.

Part 1. Defining Correspondences

First, I select two images of faces. On the left, is Ryan Campbell (me!) and on the right is George Clooney. Importantly, we are both similar sizes in the frame and facing similar angles. This will provide a smooth morph and clearer "Mid-way Face" down the line.

Next, I define correspondences between the two images, as well as computing the Delaunay triangulation of their respective points. This gives us triangles to match for each image. I chose 32 correspondence points on each face, which should be plenty.

Part 2. Computing the "Mid-way Face"

In order to compute the "Mid-way Face", we (1) compute the average shape, (2) warp each face into that shape, and (3) average the warped pixel values together (cross-dissolve the image colors).

The first step, computing the average shape, is easy. Simply average each correspondence point together. That is, compute p_mid = p_1 * 0.5 + p_2 * 0.5 for each correspondence point pair p_1, p_2. The following shows this process, with the triangulation added as well.

Next, we must warp each face into the newly calculated shape. In order to do this, we consider each of the triangles mentioned previously. Our goal is to map each triangle to its corresponding "mid-way" triangle. We can calculate the affine transformation between each triangle and use it to perform an inverse warp. We use an inverse warp because it leaves no gaps in the images, using interpolation to fill in anytime a target point is in between pixels.

The following shows the result of warping each image. The change is very noticeable around the eyes and general face shape.

Finally, now that the images have been warped to match the "mid-way" shape, we can cross-dissolve the color. That is, compute F_midway = F_warpedryan * 0.5 + F_warpedgeorge * 0.5. This yields the following "Mid-way Face":

The entire process to create the above face can be summarized as shown below:

Part 3. The Morph Sequence

In order to generate a morph sequence, instead of creating a mean face, we can use multiple different ratios of warping and cross-dissolving. We can do multiple ratios starting from 0 (original Ryan) and going to 1 (original George). With finer increments, we get a smoother video. The following morph gif is 50 frames, with the warp_frac and dissolve_frac being split equally 50 times within [0, 1]. As can be seen, this is sufficient to produce a smooth gif.

For fun, I also produced a morph sequence with me and a minion. This didn't turn out quite as well as the correspondence points were not as clear as they were for me and George. To try to make up for this, I did 100 frames instead of 50, but the worse correspondences still led to an overall worse result than the Ryan-George morph.

Part 4. The "Mean face" of a population

For the next part of this project, I utilized the FEI Face Database. This is a Brazilian database made of face images from 2005-2006 from the Artificial Intelligence Laboratory of FEI in São Bernardo do Campo, São Paulo, Brazil. I utilized 400 photos taken from 200 people, mainly represented by students and staff at FEI, between 19 and 40 years (100 men and 100 women). There are two groups of photos (no smiling and smiling), of which each person has one photo in.

The following are some example photos. Note that for this project, I actually only used the spatially normalized versions of each face, as these faces had correspondence points labeled consistently with the rest of the images.

To see an example of correspondences labeled for one of the above, see below. The FEI dataset defines 46 correspondence points for each face, but I artificially add 4 more in the corners to get a total of 50 correspondence points.

Averaging together each groups' (no smiling and smiling) correspondence points, warping each face into that average, and then cross-dissolving over each face yields the following "Mean Face" of each population:

The following are some examples of images in the dataset being warped to the average correspondence points for their respective group.

Next, I can warp myself into the average face, as well as the average face into me.

Part 5. Caricatures: Extrapolating from the mean

Next, I can use the mean face calculated from the previous step to extrapolate my own face to create a caricature that is more 'ryan-like'. The intuition goes as follows: let F_R and F_M be the Ryan face and mean face respectively, then F_R - alpha * (F_M - F_R) is an extrapolation from the mean.

In order to actually use this intuition, we calculate the extrapolated correspondence points according to the expression R - alpha * (M - R), where R is the points for Ryan and M is the mean face points. Then, we warp Ryan to these extrapolated points. The following shows this carried out for various values of alpha.

Visually, these results seem to indicate that I have a larger nose and forehead than the mean face (yikes!), as both of these features seem to expand as alpha increases.

Bells and Whistles

Here is a morphing music video of myself through the ages: "The Aging Process".