Distant Viewing with Deep Learning: Part 4

In Part 4 of this tutorial, we extend deep learning models to deal with a specific type of object detection: the localization and identification of faces.

Step 13: Python modules for face detection

We need to reload all of the Python modules we used in the Part 1.

In [1]:
%pylab inline
import collections

import numpy as np
import scipy as sp
import pandas as pd

import importlib
import os
from os.path import join
from matplotlib.colors import rgb_to_hsv
Populating the interactive namespace from numpy and matplotlib
In [2]:
import matplotlib.pyplot as plt
import matplotlib.patches as patches
plt.rcParams["figure.figsize"] = (8,8)

Let's load and look at the third corpus we have provided. It contains still images from two episodes of the sit-com Bewitched.

In [3]:
bewitched = pd.read_csv("meta/bewitched.csv")
bewitched.head()
Out[3]:
filename title frame time series season episode date
0 bw-s03-e03-mother-its-the-middle-of-the.jpg Mother, it's the middle of the night. 1199 40.4 Bewitched 3 Witches and Warlocks Are My Favorite Things 1966-09-29
1 bw-s03-e03-thats-the-proper-time-for-us.jpg That's the proper time for us, you know. 1259 42.4 Bewitched 3 Witches and Warlocks Are My Favorite Things 1966-09-29
2 bw-s03-e03-well-what-is-it-i-thought.jpg Well, what is it? I thought you should know... 1393 46.9 Bewitched 3 Witches and Warlocks Are My Favorite Things 1966-09-29
3 bw-s03-e03-that-tomorrow-has-been-chosen-as.jpg that tomorrow has been chosen as the day of th... 1499 50.5 Bewitched 3 Witches and Warlocks Are My Favorite Things 1966-09-29
4 bw-s03-e03-coven-why-who.jpg Coven? Why? Who? 1614 54.4 Bewitched 3 Witches and Warlocks Are My Favorite Things 1966-09-29

To run the code in this notebook from scratch, you will also need the face_recognition module for working with neural networks. This are not included in the default Anaconda Python installation and need to be installed seperately. The code below checks if you have face_recognition installed. If you do, it will be loaded. Otherwise, a flag will be set so that the code below that requires the module will load the pre-loaded data.

In [4]:
if importlib.util.find_spec("face_recognition") is not None:
    import face_recognition as fr
    fr_flag = True
else:
    fr_flag = False

If you are struggling with installing these, we are happy to assist. You'll be able to follow along with keras, but will not be able to apply the techniques you learned today to new datasets without it.

Step 14: Face detection

Our last set of steps concern the location and identification of faces in an image. While faces are particularly interesting, similar models exist for detecting the location of other objects using neural networks.

As an example of what this new corpus looks like, here is an image of Darrin and Samantha in their living room at the start of the episode "Witches and Warlocks Are my Favorite Things".

In [5]:
img_path = join('images', 'bewitched', bewitched.iloc[200]['filename'])
img = imread(img_path)
plt.imshow(img)
Out[5]:
<matplotlib.image.AxesImage at 0x11384dd68>

There are two faces in this image, which we can detect using the face_recognition algorithm.

In [6]:
if fr_flag:
    faces = fr.face_locations(img, 1, model="cnn")
else:
    faces = [(172, 500, 254, 418), (81, 346, 179, 248)]
    
print(faces)
[(172, 500, 254, 418), (81, 346, 179, 248)]

The output indicates that two faces are detected, and the numbers gives the coordinates for the faces known as bounding boxes. We can plot them in Python with the following snippet of code.

In [7]:
fig,ax = plt.subplots(1,1)
plt.imshow(img)
n, m, d = img.shape
for face in faces:
    rect = plt.Rectangle((face[3], face[0]), face[2] - face[0], face[1] - face[3],
                         edgecolor='orange', linewidth=2, facecolor='none')
    ax.add_patch(rect)

And, as hoped, the detected faces line up with the two characters in the frame.

Step 15: Face identification

In addition to detected where a face is, we also want to determine who the face belongs to. In order to do this, we again make use of a pre-trained neural network that returns a sequence of numbers. Just as in Step 11 with image similarity, we assume that faces of the same person are identified with similar sequences of numbers.

To illustrate how this works, lets take a set of four faces from Bewitched. The first two are of the same character (Samantha) but the second two are of of Larry and Darrin, respectively.

In [8]:
plt.figure(figsize=(14, 14))

for id, index in enumerate([145, 300, 420, 707]):
    plt.subplots_adjust(left=0, right=1, bottom=0, top=1)
    plt.subplot(1, 4, id + 1)

    img_path = join('images', 'bewitched', bewitched.iloc[index]['filename'])
    img = imread(img_path)
    plt.imshow(img)
    plt.axis("off")

We can compute the 128-dimension number associated with each face using the function fr.face_encodings applied to each face.

In [9]:
if fr_flag:
    embed = []
    for id, index in enumerate([300, 145, 420, 707]):
        img_path = join('images', 'bewitched', bewitched.iloc[index]['filename'])
        img = imread(img_path)

        f = fr.face_locations(img, 1, model="cnn")
        e = fr.face_encodings(img, known_face_locations=[f[0]])
        embed.append(e[0])
else:
    embed = np.load(join('data', 'face_test.npy'))

Using the first image of Samantha as a baseline, look at how close each of the other three images are to it.

In [10]:
embed = np.array(embed)
np.sum((embed - embed[0, :])**2, 1)
Out[10]:
array([0.        , 0.20154738, 0.70860072, 0.71007563])

The other image of Samantha is just 0.202 away but the images of the two male characters are 0.709 and 0.710 away. Using a cut-off (around 0.35 works well), we can identify images of Samantha with a reasonably high accuracy.

Step 16: Faces at scale

Finally, let's apply our face detection algorithm of the entire corpus of Bewitched iamges. The face detect is fairly slow (it took about 30 minutes on my small MacBook to do all of the images), so we recommend you set process_new to False and load the faces in from the save file.

In [11]:
process_new = False

if process_new:
    embed = []; output = []
    corpus = pd.read_csv(join("meta", cn + ".csv"))
    for index, row in corpus.iterrows():
        img_path = join('images', cn, row['filename'])
        img = sp.misc.imread(img_path)
        faces = fr.face_locations(img, 1, model="cnn")
        output.append(faces)
        enc = fr.face_encodings(img, known_face_locations=faces)
        embed.append(enc)
else:
    faces = np.load(join("data", "bewitched_faces.npy"))
    embed = np.load(join("data", "bewitched_embed.npy"))

We will compare each of the embeddings to first image used in Step 13, storing the distance in the array samantha_dist.

In [12]:
samantha = embed[145][0]

samantha_dist = np.ones(bewitched.shape[0])
for item in range(embed.shape[0]):
    for em in embed[item]:
        samantha_dist[item] = np.sum((samantha - em)**2)

Using a cut-off of 0.35, how many frames contain an image of Samantha?

In [13]:
np.sum(samantha_dist < 0.35)
Out[13]:
119

It looks like a total of 119 images show a definitive image of Samantha. Using a similar block of code to the similarity metrics used throughout these notes, what images contain the most similar Samantha faces to our prototype image?

In [14]:
plt.figure(figsize=(14, 24))

sam_index = np.argsort(samantha_dist).tolist()
for ind, i in enumerate(sam_index[:24]):
    plt.subplots_adjust(left=0, right=1, bottom=0, top=1)
    plt.subplot(8, 3, ind + 1)

    img_path = join('images', 'bewitched', bewitched.iloc[i]['filename'])
    img = imread(img_path)
    plt.imshow(img)
    plt.axis("off")