Article summary of Perceptrons by Van der Velde - Chapter

The purpose of this article is to provide an overview of how perceptrons classify patterns, and to highlight the importance of squashing functions such as activation functions and the learning opportunities of perceptrons.

Basic principles of perceptrons

A perceptron is a neural network in which neurons from different layers are connected to each other. A basic perceptron is shown in Figure 1 of the article. The network consists of two input neurons (x and y) and one output neuron (U). In general, perceptrons can have multiple input neurons and multiple output neurons. The output of neuron x is given by its output activation which is also indicated by x (so x can refer to the name of a neuron or to its output activation). The output of neuron y is its activation y.

Normally a neuron transforms the activation it receives by using an activation function (AF). The input for the AF is generally given by (total input - Activation Threshold).

Activation function ('squashing function')

In general, an activation function AF is a so-called 'squashing function'. Two important squashing functions are:

• logistic function

• hyperbolic tangent function

These functions can be seen in Figure 2 in the article.

The input can vary over a long interval (from strongly negative to strongly positive). However, the output is limited, either between 0 and 1 or between -1 and 1. This is an important feature of activation functions. It shows that the logistic function and the hyperbolic tangent function are both squashing functions in the sense that they reduce ('squashing') a (potentially) large input to a relatively small output.

The 'squashing behavior' of both functions means that the most important input values ​​are those that are around 0 (around the threshold value). For both functions you can manipulate the steepness with which the function values ​​change around 0. Moreover, the logistic function only produces positive output activations.

The logistic function

The formula of the logistic function is:

Lf(x) = 1/1 + e^-x

In this formula, the following applies:

• Lf(x): logistic function

• x: input variable

• e^-x: exponential function

Figure 3 shows e^x. The behavior of e^-x is the opposite. When x is strongly positive, e^-x is small. The logistic function (Lf(x)) therefore approaches 1 for a strongly positive input. When x is strongly negative, e^-x is large. The logistic function (Lf(x)) therefore approaches 0 for a strongly negative input. When x = 0, it applies that e^-x = e⁰ = 1, so Lf (x) = 0.5. This means that the logistic function is 0.5 with zero input.

The hyperbolic tangent function

The formula for the hyperbolic tangent function tandh (x) is:

tanh(x) = e^x - e^-x / e^x + e^-x

Both e^x and e^-x play an important role in this. When x is strongly positive, e^x is large and e^-x small. The hyperbolic tangent function tanh(x) therefore approaches 1 for a strongly positive input. When x is strongly negative, ex is small and e^-x is large. So the hyperbolic function approaches -1 for a strongly negative input. When x = 0, it applies that e^x = 1 and e^-x = 1. The hyperbolic function tanh(x) = 0 at zero input.

Threshold value

Activation neuron = AF (total input - Activation Threshold). Activation Threshold is indicated by θ. When θ > 0, a larger positive input activation is required to obtain an output of 1. It also applies that if θ <0 a larger negative input activation is required to obtain an output of 0. This is shown in Figure 4 of the article.

Classification in perceptrons

This paragraph illustrates how perceptrons can classify patterns. It also illustrates why squashing functions are such important activation functions in neural networks. Figure 5 of the article shows how the two-layer network of Figure 1 can be used to classify patterns in line with both the logistic AND-function. The figure shows the (x, y) patterns (0, 0), (1, 0) and (0, 1) that are classified as 0, and the pattern (1, 1) that is classified as 1. These points are called the 'input space'.

Classification of the AND-function can be achieved if we draw a line in the input space that separates points (0, 0), (1, 0) and (0, 1) from point (1, 1). Then all (x, y) inputs are classified to the left of the line. These are labeled as 0. All (x, y) inputs located to the right of the line are classified as 1.

EXOR classification

The EXOR problem is introduced in Figure 8 and is as follows:

x y xERORy

1 1 0

1 0 1

0 1 1

0 0 0

However, this problem cannot be classified in a two-layer network. It therefore cannot be classified with a perceptron. The reason for this is that the problem is not linearly separated in the (x, y) space. The fact that problems such as EXOR are not linearly separated was an important discovery in network theory. The solution is to solve the problem with a three-layer network. This is shown in Figure 9. In this, there is a 'hidden' layer (neurons a and b) within the input (x, y) layer and the output neuron U. The hidden layer performs two necessary intermediate classifications.

Learning with perceptrons

Figure 11 illustrates learning in a two-layer network (perceptron) that learns to classify patterns with the logistic AND- and NOR-rules. The learning procedure depends on the difference between the output of a neuron on each given example U and the output that the network needs to have a correct classification (the desired output D). The difference D-U is therefore a measure of the error that the network makes. Learning procedures that use the measure error are called 'supervised learning procedures'.