AI Applications and “Black Boxes”: How to Make Use of Recent Research on Artificial Neural Networks
Before the days of modern neuroscience, everyone from doctors to philosophers had their own theories concerning the complexities of the human brain. Even today, understanding the brain’s processes is a daunting challenge, with many infinitesimally small and complex neural chain reactions. There are 100 billion neurons in the human brain, each with 1000 or more connections to other neurons, resulting in a network 100 trillion strong.
Replicating this type of interconnectivity is just part of the challenge when developing artificial neural networks. The human brain may long have been surpassed by computational processing power for some tasks, such as basic mathematical applications, but in areas that require more complex visual recognition and reason, artificial systems lack the finesse of the human brain.
“[Neural] networks have been described as ‘black boxes’, as a reference to how inaccessible they are”
This is beginning to change, though, as synthetic neural networks have shown themselves to be increasingly adept at image recognition. However, understanding how these networks reach their conclusions is difficult, as their processes are hidden beneath layers of algorithmic programming. Indeed, these networks have been described as ‘black boxes’, as a reference to how inaccessible they are.
In recent years, there have been key breakthroughs in neural networks that have provided researchers with new techniques to understand the inner workings of these black boxes. Interestingly, these techniques are reminiscent of pioneering techniques used in the field of neuroscience, which were crucial in gaining a fundamental understanding of how brains function.
These techniques are important, as they potentially hold the key to improving on deep neural network designs, thereby paving the way for new levels of artificial intelligence.
BDJ spoke with Anh Nguyen, Assistant Professor of Computer Science at Auburn University, and a leading figure in neural network research, to learn more about techniques and their limitations.
Breaking the Black Box
Decoding the inner workings of neural networks has historically been a challenge. When humans look at a dog, for example, our brain processing the image is completely removed from our consciousness.
Light enters our eyes, hits our retinas and signals are carried along the optic nerve to the brain, where the image is decoded. With artificial neural networks, image recognition is all digital, with all the levels of computation taken into account and mapped.
“A key challenge is these intermediate digital formats are still not interpretable to humans”
“A digital photo is often represented as three sheets (or matrices) of numbers, each for an RGB color channel,” Anh explains. “Humans could naturally make some sense out of a photo (e.g. of a cat) but would have a hard time interpreting the corresponding three matrices of numbers.
“Say, we have a typical artificial neural network e.g. that outputs a label ‘dog’ for a given image of a dog. This is a machine that takes in the above three sheets of numbers and outputs the label ‘dog’. However, internally, the input sheets of numbers get transformed and resized smaller from one ‘digital image” to another, many times before being transformed into the final output ‘dog.’
“A key challenge is these intermediate digital formats are still not interpretable to humans. Also, the internal, complex transformation and resizing tricks (that the machine designed by itself) are also represented by a bunch of numbers, which humans don't know how to naturally interpret.”
Feature Visualisation and Activation Atlases
Feature visualisation has offered researchers a route to understanding the black boxes of artificial neural networks. The technique is a modern continuation of groundbreaking work carried out by David Hubel and Torsten Wiesel in 1959, which sought to better understand how brains react to visual stimulation.
In these experiments, which would be extremely controversial by today’s standards, wires that measured neuronal activity were implanted into cats’ brains. Hubel and Wiesel then projected various images onto a screen for the cats to watch, in order to detect which images stimulated the neurons.
At first, none of the various images provoked any kind of neuronal activity. However, when one of the slides got jammed in the projector, one neuron in one of the test subjects’ brains began firing. The two researchers eventually discovered that this firing neuron was responding to the sharp shadow on the screen from the edge of the jammed slide. This neuron, therefore, was responsible for recognising hard lines/edges in an image.
This notion that individual neurons are triggered by specific visual components has been a foundation for subsequent neural research, including the artificial neural networks of today. In recent years, deep visualisation – also known as feature visualisation or activation maximisation – has built upon the Hubel and Wiesel experiments.
“Feature visualisation is the utilisation of a pre-trained, neural network that classifies images, with the idea being to synthesize a set of input images that all cause an individual neuron inside the network to fire strongly,” Anh says. “By looking at this set of images, we gain further insights into what the neuron is doing. For example, we've learned that there are neurons that fire for ‘mosque’ or ‘lipstick’ (Fig. 1; top left).
“Feature visualization can paint an image for each neuron in a neural network to provide a holistic, visual interpretation of what the neuron is doing. For example, via such techniques, we were able to find out neurons that automatically learn to detect eyes, noses, and mouths, which collectively help the entire neural network in detecting faces.”
The example above shows the individual neurons in a single layer of the network responding to individual portions/aspects of the input image, such as the outline of the man’s hair, or the hard edges between the t-shirt in the foreground and the wall in the background.
This avenue of research has been hugely useful for researchers, but there have been recent expansions on the technique.
“While activation maximisation only paints pictures to describe one single neuron, activation atlases paint pictures to describe multiple neurons at the same time. That is, it tries to provide the concepts or thoughts captured by a group of neurons,” Anh says.
Activation atlases have been developed by researchers from Google and OpenAI. They are an important progression because they are not limited to the stimulation of a single layer of a neural network. Instead, individual neurons are targeted to see what excites them, moving on to clusters of neurons, and then neurons within different layers of the network.
As this process goes on, researchers have been able to map out the visual activations of these different clusters and layers of neurons. In doing so, they are creating a psychedelic map of the neural network.
Activation atlases are a huge step towards understanding the functioning of neural networks because they provide a tangible, visual representation of the associations and interactions between different clusters of neurons.
Researchers and developers are afforded a view of the bigger picture, and are able to hone in on common combinations of neurons. “Thanks to these tools, we now have a better understanding of what these internal neurons are doing,” Anh said.
“These insights are sometimes valuable in designing a higher-performing neural network. Sometimes, we find neurons that seem to be behaving in some unexpected ways which may reveal a bug or serious defects in the decision-making logic of the network.
“For example, deep visualisation actually helped me and other researchers reveal what we call ‘adversarial or fooling examples’ i.e. the input images that cause neural networks to misbehave (e.g. labelling TV static "lion" with certainty confidence).”
The discovery of these adversarial influences that can be utilised to fool neural network image classifications will pose a real challenge to developers.
A fundamental aspect of activation atlases is that the real-world objects represented by the clusters are grouped together by their visual similarities to one another. A golf cart can, therefore, be a few activations away from a sewing machine, while a dog’s face can be close to a bunch of flowers.
Anh has focused on this as part of his research and discovered that a simple act of rotating a source image can greatly impact its classification by the neural network. For example, rotating a school bus slightly could alter its classification to ‘punching bag’.
Examples like the one above can be categorised as honest mistakes by neural networks thanks to the unpredictable manipulation of objects into unnatural positions (training data examples of school buses aren’t too likely to include many instances of the bus veering upwards off the road at 45 degrees).
However, as AI programs develop, and these programs are reliant on high-quality training data, Anh points out that there is the potential for nefarious individuals to deliberately attempt to affect how they classify inputs. Anh calls these “adversarial or fooling examples.”
“Recent research has shown that one can stick fooling patches on a STOP sign and cause a self-driving car to keep going thinking that was a Speed Limit sign”
“Recent research has shown that one can stick fooling patches on a STOP sign and cause a self-driving car to keep going thinking that was a Speed Limit sign. Researchers were also able to fool face-recognition-based security systems by painting their glasses with fooling patterns (i.e. strangers could fool your security camera and get access to your house with these glasses). In general, any AI system could be fooled if an attacker is allowed to change the input of the system.
“This is a hard problem that there is currently no general solution for. At the moment, to combat this issue, the most effective method is to train the AI on a lot more data and harness various types of input sensory sources. For example, a self-driving car nowadays often will not rely only on RGB images, but also fuse the inputs from a suite of other sensors such as LiDAR, Radar, GPS, IMU, etc to make decisions.”
The Issue of Human Interpretation
Moving forward, the breadth of AI applications that rely on artificial neural networks to provide image and physical object recognition is set to increase. Feature visualisation and activation atlases are acting as tangible visual representations of the processes behind interpreting images.
Adversarial exploits add to the concern felt by many that artificial intelligence is lacking any kind of direct consensus on how to address foundation-level vulnerabilities. AI is awkwardly placed in a sense – it is designed to display processing and complexity potential that is far beyond the capabilities of any human, yet it is expected to display human-like reasoning in many instances.
Unfortunately, the logic displayed by humans is often influenced by underlying bias and other unconscious leanings.
“It is an inherent part of the definition of "interpretation," Anh says. “That is, we want to explain the AI behaviors in a way that is understandable to humans. However, AI behaviours could be very complex beyond our explainability/understanding. This is similar to the fact that humans do not know how their brain actually works internally when making a decision. However, humans can only rationalise their decisions.”
“We want to explain the AI behaviors in a way that is understandable to humans, however, AI behaviours could be very complex beyond our explainability/understanding”
The complexity of AI naturally poses challenges for understanding its fundamental cognitive foundations. Simple image recognition through neural networks is being mapped by the above methods in ways that displays logical visual grouping.
However, as AI progresses, we are more and more likely to encounter instances of its complex behaviours that make no apparent sense. In this regard, there is no real alternative option other than continuing to develop outlets such as activation atlases which allow researchers and developers to explain AI’s reasoning in ways that feel more human.
One of the key problems with the development of neural networks is that the barrier to entry into working with them is high. They need huge datasets to learn from, are incredibly complex and, crucially, have to be quite large. A pair of researchers at MIT, though, have discovered a way to create artificial neural networks that are around a tenth of the size but possess the same computational power. It is a breakthrough of some significance, opening the door for other researchers to create smaller, faster AI programs that are just as smart as those that exist already.
The researchers were able to reduce the size by introducing regular ‘pruning’ – the process by which our brain regularly trims old, unused neural connections from memory – during training. The result was networks that were far smaller, had the same computational power and were actually more accurate than those which pruned only once at the end of training. The future of these smaller neural networks is an exciting one, given that they could become small enough to be used in small electronic devices, a potential lightbulb moment for the burgeoning IoT industry.
Illustrations by Kseniya Forbender
To contact the editor responsible for this story:
Margarita Khartanovich at [email protected]
- How Vulnerable Is the Internet of Things (IoT) to Hackers?
- Blockchain Interoperability Remains a Critical Missing Puzzle Piece
- How to Keep A Business Safe: Tips From IBM and Shapeshift.io Cybersecurity Experts
- China Vs the US: Who Will Win the AI Race?
- AI Applications and “Black Boxes”: How to Make Use of Recent Research on Artificial Neural Networks