When Artificial Intelligences need to make decisions, they can easily fall victim to hostile attacks. How easy it is for an attacker depends on the shape of the boundary with which the neural network separates the possible decisions from each other. In our publication “Heating Up Decision Boundaries”, we measure the heat emitted by such a boundary and deduce its shape from it.
Neural networks are deeply woven into our daily lives nowadays, performing numerous tasks with a precision unattainable for humans. It is all the more surprising that they can be victims of attacks that a human would never fall for.
In this example, both human and machine would agree on the left image – it is a pig. However, adding a certain noise convinces the machine that it is looking at an airplane, while the human eye sees no difference between the images. What happened here?
The shortest path to the decision boundary
First, we need to understand how neural networks categorize images. In this example, the network classifies all handwritten digits that lie in the lighter gray area as twos and those in the darker gray area as sevens. The location of a digit is determined by the brightness of its pixels. Starting from a two, we could change individual pixels until our image shows a seven. As we traverse this path and ask the neural network how it categorizes our image, there must be a point at which it changes its judgment. We call this point the decision boundary; in the image above, it’s the white border between the lighter and darker gray areas.
Adversarial algorithms seek the shortest path to the decision boundary to change the neural network’s judgment by minimal manipulation of the image. In the above example, such a strategy is unlikely to succeed because the boundary is approximately smooth and always maintains a large distance between the sevens and twos. In this case, we call the network robust. The following network, however, is extremely vulnerable.
The hostile algorithm can exploit the spikes in the decision boundary to its advantage and crosses the boundary with minimal manipulation of the data. Such a spike seems to have been present in the above example with the pig, explaining why the geometry of decision boundaries is at the center of a lively discussion about the vulnerability of Artificial Intelligence. However, in realistic scenarios, gaining insights into this geometry is extremely challenging, prompting the constant search for new and efficient methods to sharpen our understanding of neural networks.
Visualizing decision boundaries
We propose such a method for determining the geometry of boundaries in our new ICLR publication “Heating Up Decision Boundaries: Isocapacitory Saturation, Adversarial Scenarios, and Generalization Bounds” by our Lamarr researchers Bogdan Georgiev, Lukas Franken, and Mayukh Mukherjee (Indian Institute of Technology Bombay). The idea is to interpret the decision boundary as a heat source and translate the amount of emitted heat into geometry: the process can be imagined like old-fashioned radiators, whose angular and pointed bodies try to maximize the heat-emitting surface. Following the same principle, the decision boundary also emits more heat when it is sharp and uneven.
But how can we simulate this heating? We can interpret heat physically: What we feel as warmth on our skin is actually just a multitude of tiny particles colliding with the skin – less heat, fewer collisions, and vice versa. We can also use this intuition to represent the heating of decision boundaries. So, we represent the heating by randomly moving particles emitted from the decision boundary and spreading in space. We use the training examples as measurement points – if they receive many particles, the decision boundary heats up strongly; otherwise, less.
However, another problem arises here because we don’t even know where this boundary lies. All we know is where the examples, i.e., the twos and sevens, are. In practice, we measure how much heat the boundary receives from the example’s heating, not vice versa. The analysis remains identical because the two roles – heat emitter and heat receiver – are interchangeable. This fact is formalized by Feynman-Kac duality. The proportion of particles colliding with the decision boundary during their movement corresponds to the emitted heat. With this method, we can not only detect whether the decision boundary exists but also whether it is flat, round, or sharp. This distinction was not possible with any of the known methods. In fact, the research community has so far been convinced that robust networks emerge when the decision boundaries are flat. We demonstrate that this is not the case. Locally, robust networks also have sharp surfaces, so their robustness seems to be generated by other difficult-to-detect properties. In this sense, we deepen our understanding of the vulnerability of neural networks.
Implications for the generalization ability of neural networks
Indeed, resilience against adversaries is not the only property influenced by the geometry of decision boundaries. We also show that geometry provides clues as to how well a neural network generalizes. The ability to generalize is the main reason for the usefulness of Machine Learning. It means, for example, that we can show a network example of handwritten threes and then it is able to recognize threes, even though they are not the threes from the training set. If the network could not generalize from the learned examples, it would not be able to recognize similarity but only whether an example is exactly identical to a seen example. It turns out that the ability to generalize well is more likely when decision boundaries are a) flat and b) far away from the training examples. Our method offers opportunities to gain insights into both properties.
How decision boundaries are shaped in vulnerable or robust neural networks has been an active research topic for several years. Our technique of heating decision boundaries and inferring their shape from the amount of emitted heat enriches the current state of knowledge on this topic: Even robust networks have locally sharp and jagged surfaces. There are even more properties that can be determined by the geometry of decision boundaries. These include, for example, the overall quality of classification or the number of parameters that play a role in the network’s functionality. In future projects, we will investigate these relationships more closely.
More information in the associated publication:
Heating up decision boundaries: isocapacitory saturation, adversarial scenarios and generalization bounds B. Georgiev, L. Franken, M. Mukherjee, ICLR, 2021, PDF.
