Finding the faces

• note: this is a work in progress...

Finding the faces...

With the sandscope instrument hardware in good working order, we can now focus on refining the software that drives it, and execute the search for faces among millions and millions of sand particles.

5 focal layers

Highly magnified images have a shallow depth of field. The sand particle depicted above has a diameter of about 0.5 mm. Each photo has been taken at a slightly different distance from the object, and thus the focus is on different areas of the particle. To increase the chances of face detection, the detector will have a look at all the focal variants of the same object. Currently 8 'focal layers' are used. The first and last shots of a focus series are usually totally fuzzy, the remaining ones have varying degrees of sharpness. Only a few will have truly sharp areas. The latter are the targets that are worth examining, the others can be skipped, even before the face detector will consider them. The processing pipeline is thus set up as a two-stage filter: first sharpness, then face detection. Only samples that pass both filters make it to the output. Both filters will be implemented as artificial neural networks.

The reference face detector

The off-the-shelf face detector used for this project is a Go implementation of the PICO algorithm described in the paper "Object Detection with Pixel Intensity Comparisons Organized in Decision Trees"
by Nenad Markuš, Miroslav Frljak, Igor S. Pandžić, Jörgen Ahlberg, Robert Forchheimer

https://arxiv.org/abs/1305.4537

PICO is an efficient detector that compares favorably with some industry standard face detectors. It will be the baseline reference to compare our own custom detectors with. PICO returns a score for each detected face, which allows sorting of the search results. Unfortunately, an image with a higher PICO score does not necessarily  look more like a face than an image with a lower score, at least when judged by a human. It has happened that we were unable to see a face in an image with a fairly high PICO score.
The main research task is thus developing a detector that reliably detects faces and gives them face-ness scores that we humans agree with. This will then enable the Sandscope instrument to maintain a top-100 or top-1000 of best faces ever found, without any human intervention.

25 'faces' (1 day search)

These are the results of the first 24-hour test run: the 25 sand particles that more or less look like faces.
I agree with the detector in some cases, in other cases, i wonder what it was triggered by. There is obviously still a lot of work to do on the face detector, but there is reason for optimism: the above image is proof that the Sandscope instrument can actually be implemented, and is not just a theoretical device: faces -albeit vague ones- do appear in the output!
The PICO face detector is still way too lax when it comes to accepting a particle as a human face. This test run produced a collection of about a thousand candidates. Increasing the minimum detection score needed to accept an input image as a 'face' will result in fewer candidates, but doing so will also reject many 'good' sand faces. These 25 images were selected by a human, and they did have a wide variety of scores. So unfortunately PICO's opinion about a clear face does not match my own human opinion about what is a clear face.
Note that the PICO detector tends to favor 'potato heads'. It likes round and blurred shapes with vague facial features. This is likely a result of the inner workings of the algorithm, the way it models an average face as sets of gradients?

Facebook is interested to know who this is!

This grain of sand managed to fool the Facebook face detection algorithms. Many more will follow, it is simply a matter of dedicated search! The Facebook and Sandscope detectors agree on the face-ness of this image: both detectors experienced a level of stimulation that triggered a response. 
Interestingly, when looking at a result like this, once you see the face, you no longer see the grain of sand. Once the image is interpreted in one way, it is hard to switch to another way. 

The Artificial Neural Network face detectors

The custom detector is a regularly formed convolutional network, which has proven to be quite good at image classification in earlier experiments. The input is a 128x128 grayscale image, the output is a class probability. Tests are being done with a 2-class (face/no-face) and a 4 class (none/bad/ok/good) classifier.

The layers of the network are organised as follows:

  convolution       f1 128x128          (3x3 kernel, with dropout)
  max pooling       128x128 → 64x64
  convolution       f2 64x46            (3x3 kernel, with dropout and batch normalization)
  max pooling       64x64 → 32x32
  convolution       f3 32x32            (3x3 kernel, with dropout and batch normalization)
  max pooling       32x32 → 16x16
  convolution       f4 16x16            (3x3 kernel, with dropout and batch normalization)
  max pooling       16x16 → 8x8
  convolution       f5 8x8              (3x3 kernel, with dropout and batch normalization)
  max pooling       8x8 → 4x4
  convolution       f6 4x4              (3x3 kernel, with dropout and batch normalization)
  max pooling       4x4 → 2x2
  fully connected   u1
  fully connected   u2                  (output)

f1 to f6 are the numbers of feature maps of the convolution layers, and u1 and u2 are the number of units of the fully connected layers. Experiments are performed with a variety of settings for these numbers, from small to medium to large sized networks.

The detector is implemented using python and TensorFlow, and runs as server process that uses shared memory for the input images, and a simple text based API over UNIX domain sockets to receive requests and send replies. The main Sandscope application, written in Go, acts as a client of this 'isface' server. This setup turned out to be efficient and reliable. The image processing is reduced to a minimum, no encoding or decoding is necessary to perform the classification. The memory footprint is low, as all the image memory is re-used.

The training data

The custom 2-class (face/noface) networks were initially trained on images containing human faces, and on images that don't contain faces. There are various existing datasets available for download. The following two have been used:

1. Labeled Faces in the Wild : http://vis-www.cs.umass.edu/lfw/index.html
   
   13233 color images, 250x250 pixels, with the faces of 5749 people, gathered from the web. The faces have been centered.
2. CelebA : http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
   
   202599 color images, 178x218 pixels, with the faces of 10177 people, gathered from the web. The faces have been centered.

These face images were first made square (extracting the center region), then converted to grayscale and resized to 128x128 pixels.

The no-face images were made by automatically downloading random images from wikimedia, resizing them to about 1M pixels, and extracting random crops of 128x128 pixels. The images from wikimedia were manually checked to see if they contained faces. If they did, they were deleted.

It turned out that the above two face image datasets don't contain any young people, nor many old people. The majority is white. To reduce this bias, we gathered images from the web of babies, children, elderly people and all sorts of non Caucasian faces. We also collected faces found in drawings, paintings and sculptures. The faces in artworks are not photo-realistic faces, but very interesting nonetheless, because our human perception has no trouble detecting them as faces.

First experiments

The first neural network training sessions went very smoothly, the reported training and validation accuracies were impressive. Even networks with less than 100K parameters are able to reach validation accuracies of 99.8% and above. That means only 2 mistakes per 1000 input images! And it was funny to see that most of the mistakes made were actually faces of teddy bears, cats and dogs. 

generation 1 neural network, 1 day search

See here a human-made selection of the 25 most face-like sand particles that were found in a 1 day search. The detector gave a large amount of suggestions, in this case about four thousand, of which only a few turned out to have sufficient face-like qualities for us humans. Note that the neural network detector came up with face suggestions that exhibit more shape and expression variety than the PICO detector (there are less 'potato heads' here). This is good.

generation 2, 2 day search

The above image shows a human-made selection of the 25 most convincing faces of the next generation network. This 2nd generation network was trained with an extended dataset that contained several thousands of false positives that the 1st generation network produced. False positives are the cases where the network classified a grain of sand as a face, where we as humans do not see a face at all: detection mistakes. 
We hoped that including such mistakes as negative feedback would make the new network more critical, in effect suggesting less suggestions that were obviously false (in the eyes of a human). This turned out to be the case. The number of face suggestions that the new network produced was slightly lower than before, and the number of actually recognizable faces was significantly higher.

Encouraged by the success of the 'negative feedback' approach, we set out on training the next generation of networks, after adding examples of the failures of the previous network to the training data set. The path forward looked to be simple: train each new generation while adding samples of the mistakes of the previous generation, and in doing so, arrive at networks that are very critical and hardly make mistakes at all any more!
Sadly, this strategy didn't have the desired effect. It turned out that the new networks were forced into a corner too much: there seemed to be a suppression of generic face features, like the overall vertical oval shape. We think that too many mistake samples did have this overall shape, but lacked any further details. So the overall shape was more and more associated with being wrong, and the networks tried to find other face shapes which did still have some details right. This resulted in many face suggestions that looked very compressed and distorted.

When confronted with sand particles -which often exhibit rather random morphological features- the artificial neural network can easily come up with a 'face-ness' score that is very different from the score we humans make. When searching for faces in sand, the system is operating in a vague area where it has to decide about all sorts of cases that it never was exposed to during training. The AI can see a face where we as humans don't see one, and vice versa. Note that even two humans don't necessarily come to the same conclusion regarding the face-ness of a sand sample.
How to align the machine and human perception more, especially along the gradient of 'not a face' to 'hardly a face' to 'plausible face' to 'convincing face'? Ideally, the detector would only offer suggestions that we as humans agree with: all clear and convincing faces. 

The first face detector networks we implemented were binary classifiers that distinguish between face and
no-face inputs. Realizing that the Sandscope instrument requires a detector with a more nuanced output than only yes or no, experiments were started with multi-class detectors that label their output as being either 'not', 'bad', 'ok' or 'good' face. This meant we had to create a training data set from scratch.

The first idea was to create a not/bad/ok/good face data set automatically, by taking an existing face data set, and using the original face images as the examples of 'good' and then gradually 'messing up' the face images more and more, to form 'ok', 'bad' and 'not' examples. Unfortunately the face detector networks turned out to become very good at discerning between these automatically made artificial samples, but failed to deliver meaningful suggestions when processing images of actual sand particles. The networks must have learned how to recognize the 'fingerprints' of the methods to used to compromise the original images, without caring about the actual underlying content (faces).

What about creating a multi-class training set that purely consists of sand particle images? After all, the Sandcope has been running for months already, and has stored images of thousands and thousands of sand face suggestions! All these images can in principle be sorted into the not/bad/ok/good sample subsets manually, using human judgement, and serve as examples. Training a detector only on sand particles might have an interesting advantage: the detection is then purely based on the pareidolia effect. The network has never seen a real face but forms an imaginary prototype of a face by being exposed to more and more examples of sand that look like faces.

At this point in time, the sand face data set is rather small. This is a problem for artificial neural networks, because they tend to perform better with larger amounts of data to train on. Furthermore, the number of samples per subset varies a lot. There are only few 'good' faces, some more 'ok' faces, quite a few 'bad' faces, and lots of 'not' faces among the suggestions that the Sandscope returns when searching the sand.
As of 2019-11-06 the data set is composed of 693 'good', 1599 'ok', 3324 'bad', 14302 'not' sand face samples. It will be important to expand the data set over time to be able to train more sensitive and efficient detectors. The idea is to train a new network, install it on the Sandscope, do a search for a few weeks, sort out the newly stored suggestions, expand the data set, do a next training, and so on.

Currently the multiclass detectors are not performing so well. They reach accuracies of around 60%, which is significantly better than the 25% which would be the accuracy of a network that makes random guesses, but one would want to do better. Is it the right approach?
We are also still training binary face/no-face detectors using the sand face data set, and these detectors perform better, reaching accuracies between 80 and 90%. But even these networks tend to return too many false positives in their face suggestions. When glancing over these suggestions, you can see why the AI sees a face in them, but too often it takes too much imagination and mind bending effort. Are there networks that are more critical, and at the same time, don't skip good faces?

note: this is a work in progress...