Are you bad with faces? Maybe a monkey could beat you.
Scientists have just worked out how to accurately reconstruct pictures of human faces by monitoring monkeys’ brains, solving a decade long mystery in neuroscience.
Professors Steven LeChang and Doris Tsao used a mix of brain imaging and single-neuron recording to measure the brain activity of macaques while showing them pictures of human faces. When measuring the brains’ electrical activity, they could see patterns, which they translated to a reconstructed image.
Tsao, and colleagues from the California Institute of Technology, had already found that six blueberry-sized areas of the brain are involved in identifying faces, using functional magnetic resonance imaging, or fMRI. All of these so-called ‘face patches’ are located in the inferior temporal cortex, which is known to be associated with visual object recognition. When they ‘see’ an object, the neurons emit electrical signals.
In their recent research, Tsao and LeChang devised an array of 50 dimensions that could be used to describe a face, including skin tone, texture, face shape and a range of facial distances. When they measured the electrical activity from the macaques’ brains, they found that these facial characteristics created a response in 205 neurons, which, when combined, allowed the macaques to understand and recognise the human faces.
“People always say a picture is worth a thousand words,” says Professor Tsao. “But I like to say that a picture of a face is worth about 200 neurons.”
The pair found that the firing rate of each neuron, which are called ‘face cells’ in the field, corresponded to different facial features along an axis, and would only change if the feature changed. The face cells were found in just two face patches, with cells in different patches processing complementary information – the appearance due to distances between features in one patch, and information about shape in the other.
“This was completely shocking to us – we had always thought face cells were more complex. But it turns out each face cell is just measuring distance along a single axis of face space, and is blind to other features,” Tsao says.
Comparing the measured neuron responses to a new face with predicted responses, LeChang and Tsao were able to develop a model for which features were observed by which neurons, and then recreate the face that the monkey was looking at. The results were remarkably similar, and you can see them yourself here.
The mechanisms behind the monkeys’ facial recognition are not yet fully understood, but the findings still challenge previous schools of thought. Other scientists in the field believe that each face cell recognises a specific type of face, despite this idea having already been challenged by the observation that even when looking at very different types of faces, some face cells always light up in the same way, but do not fire when looking at other objects, like vegetables and even other body parts.
That’s not the only reason the results were unexpected – the team originally set out to work out how to convert facial images into a numerical representation, by converting 25 measurements of a face into a matrix. They chose a scheme to mirror the responses of neurons and, as luck would have it, found that there was almost a direct mapping between the two when they took brain recordings.
“The predictions were so good, I was kind of amazed,” said Tsao. In fact, when she first saw the results she asked her colleague if he had made a mistake. They later realised that the method was the most mathematically efficient way to convert faces to numbers.
“If you look at methods for modelling faces in computer vision, almost all of them … separate out the shape and appearance,” she says. “The mathematical elegance of the system is amazing.”
Whether we understand it or not, this work could have amazing real-life applications, from inspiring new facial recognition algorithms to forensic reconstruction of a criminal’s face by analysing a witness’ brain activity.
“A face is impossible to describe in words,” says Tsao. “One can imagine applications in forensics where one could reconstruct the face of a criminal by analysing a witness’s brain activity … you could imagine reading out what face a person is imagining.”