The invention of neural networks

Neurons and weights

In machine learning, a computer learns decision criteria from examples. But how does it process the data and learn those criteria? Researchers have proposed many mathematical and statistical models. One of the most influential is the artificial neural network, inspired by the neurons in our brains. The idea was to represent aspects of how connected neurons process information using computations and connections inside a computer.

Countless neurons exchange signals to support sensation, movement, and other functions. Imagine touching a hot pan. Sensory receptors in your skin detect the heat, and signals travel through neurons. Signals reaching the muscles in your hand and arm cause them to contract, pulling your hand away.

What matters is how the neurons are connected. Some circuits are involved in moving your hand; others process sounds or visual information. Repeated experience can strengthen connections between neurons that are frequently active together.

Artificial neural networks borrow this idea. A number called a weight describes the influence of a connection. You can think of it as indicating how strongly one neuron’s signal contributes to the next neuron’s calculation.

In this diagram, darker lines represent larger positive weights and therefore stronger contributions. A neuron can connect to several others with different weights and receive input from several neurons at once.

Weighted sums

Here is how weights enter the calculation. The value inside each input neuron represents its signal strength. The receiving neuron multiplies each signal by its connection weight, then adds the results:

(0.2 × 0.8) + (0.4 × 0.4) + (0.5 × 0.2) = 0.42

This calculation is called a weighted sum.

Imagine comparing applicants using their English, science, and math scores. If math matters more for this particular program, we could multiply the English and science scores by 1 and the math score by 2. Math would then contribute twice as much per point to the final score.

Both applicants have a raw total of 250. After giving math a larger weight, applicant B, who has the higher math score, receives the higher weighted total. A weighted sum lets different inputs contribute different amounts to the result.

Neural networks as functions

We can also describe these connections using functions. In chapter 1, we saw that a function connects inputs to outputs according to a rule. Let the incoming neuron values be inputs x and the receiving neuron’s result be output y. A function f can describe the calculation connecting them.

For three inputs producing one output, we write y = f(x₁, x₂, x₃). This means that f takes x₁, x₂, and x₃ and calculates y.

Inside the function, each input is multiplied by a weight that determines its contribution. For the network above:

⇒ f(x₁, x₂, x₃) = (0.8x₁) + (0.4x₂) + (0.2x₃)

Now increase the number of output neurons to three. Each output neuron calculates its value using the inputs and the weights of its own incoming connections. We can therefore describe the network with three functions, one per output.

The functions f₁, f₂, and f₃ are different even though they receive the same inputs. Each output neuron has its own set of weights.

For example, the functions might be:

Expanding the network

The nervous system contains vast numbers of connected neurons. Our examples have been small, but artificial neural networks can also contain many more units and connections.

Let’s expand our example to n input neurons and m output neurons.

Neurons at the same stage of a network form a layer. The input layer receives the initial signals, and the output layer produces the results.

We can still express these connections as functions. We use x for inputs and w for connection weights.

With n inputs, each output neuron calculates a result from n values. With m output neurons, we have m functions, one for each result.

The network turns n inputs into m outputs.

The weight wij determines how input xj contributes to output yi. The first index, i, identifies the output neuron. The second, j, identifies the input neuron.

For example, w21 is the weight connecting the first input x1 to the second output y2.

Writing out all m functions, each with n inputs, would take a lot of space. We can shorten one function to:

Here, x groups the inputs x₁, x₂, …, xₙ, and w groups their weights. The dot · means multiply each input by its corresponding weight, then add the products. The expression f(x) = w · x is a compact way to write the weighted sum for one output neuron. The notation is shorter, but the calculation is unchanged.

For m output neurons, each function has a different weight vector: w₁, w₂, …, wₘ. Stack them into a matrix W, and we can express all the outputs y₁, y₂, …, yₘ in a single equation:

Multiplying the input vector x by W produces the output vector y. Thus, y = Wx describes how the weighted connections between two layers transform multiple inputs into multiple outputs.

To calculate y₂, use the second row of W: multiply each weight in that row by its corresponding input, then add the products. Matrix notation packages the same weighted-sum calculation into a more convenient form.

Exercises

Decide whether each statement is True or False.

1. A weight describes how a connection between two neurons contributes to the result.
2. A larger weight means the neuron itself is always more active.
3. The group of neurons that first receives external inputs is called the output layer.
4. w₂₁ is the weight connecting the first input x₁ to the second output y₂.
5. y = Wx can represent the weighted-sum calculation of a neural network layer with multiple inputs and outputs.