What's in a Face? Metric Learning for Face Characterization

Please reload

Omry Sendik, Dani Lischinski and Daniel Cohen-Or

Tel Aviv University

We present a method for determining which facial parts (mouth, nose, etc.) best characterize an individual, given a set of that individual's portraits. We introduce a novel distinctiveness analysis of a set of portraits, which leverages the deep features extracted by a pre-trained face recognition CNN and a hair segmentation FCN, in the context of a weakly supervised metric learning scheme. Our analysis enables the generation of a polarized class activation map (PCAM) for an individual's portrait via a transformation that localizes and amplifies the discriminative regions of the deep feature maps extracted by the aforementioned networks.
A user study that we conducted shows that there is a surprisingly good agreement between the face parts that users indicate as characteristic and the face parts automatically selected by our method.
We demonstrate a few applications of our method, including determining the most and the least representative portraits among a set of portraits of an individual, and the creation of  facial hybrids: portraits that combine the characteristic recognizable facial features of two individuals. Our face characterization analysis is also effective for ranking portraits in order to find an individual's look-alikes (Doppelgangers).

Face Hybrids

Our metric results in higher rankings for the Doppelgängers present in a portrait set, compared to the baseline metric. Columns 1, 5, and 9 show the change in the ranking order of the Doppelgänger portraits when switching from the baseline metric to ours. It may be seen that in nearly all cases, the ranking is improved and the Doppelgänger images are ranked higher (smaller indices) when the set is sorted according to our metric. In each block of images, the left column shows the representative portrait of the individual for whom Doppelgängers are sought, while the middle and right columns show the top ranked portrait in the set according to our metric and the baseline metric, respectively.