A network of simple parts
An artificial neural network is a mathematical structure made of many small, simple units connected together. Each unit takes some numbers in, does a basic calculation and passes a number on. On its own, a unit is almost trivial. The power comes from using thousands, millions or even billions of them in combination, and from adjusting how they connect until the whole network does something useful.
The name comes from a loose analogy with the brain, whose neurons pass signals to one another. The analogy is only a rough inspiration. Artificial networks are far simpler than biological ones, and nobody should assume they work the way a brain does.
What one unit does
Imagine a unit deciding whether a photo shows a cat. It receives several numbers as inputs, perhaps summaries of features from earlier units. Each input has a weight, a number that says how much it matters. The unit multiplies each input by its weight, adds the results, adds a small adjustment called a bias, and then passes the total through an activation function. That last step is a simple rule, such as “output zero if the total is negative, otherwise output the total,” which lets networks represent patterns more complex than straight lines.
The weights and biases are the network’s learned settings, often called parameters. Training is the process of finding good values for them.
Layers
Units are organized into layers.
- Input layer: receives the raw data, such as the pixel values of an image or numerical codes for words.
- Hidden layers: sit in between and transform the information step by step.
- Output layer: produces the result, such as probabilities for “cat,” “dog” or “neither.”
A network with many hidden layers is called deep, which is where the term deep learning comes from. Stacked layers allow the network to build up from simple features to more abstract ones. In an image system, early layers often respond to edges, middle layers to shapes or textures, and later layers to larger structures. These features are discovered during training rather than programmed by hand.
How a network learns
Learning follows a repeating cycle.
- Make a prediction. Feed an example through the network and read the output.
- Measure the error. A loss function scores how far the prediction is from the correct answer.
- Work out blame. An algorithm called backpropagation calculates how much each weight contributed to the error.
- Adjust. A method called gradient descent nudges every weight a small step in the direction that reduces the error.
- Repeat over many examples, often many passes through the full data.
A useful picture is walking downhill in fog. At each step you feel which direction slopes down and move a little that way. With enough steps you usually reach a low point, though not necessarily the lowest one.
A tiny example
Suppose a network predicts whether a person will like a film using two numbers: how much they liked comedies and how much they liked thrillers, each on a scale from 0 to 1. At the start the weights are random, so predictions are poor. Each time the network is shown a person’s known verdict, it adjusts its weights slightly. After many examples, the weights tend to reflect how strongly each genre preference predicts the verdict. Real networks work the same way, only with far more inputs, layers and examples.
Different designs for different jobs
- Feedforward networks pass information in one direction and suit many general prediction tasks.
- Convolutional networks are designed for grid-like data such as images, and underpin much of computer vision.
- Recurrent networks were designed for sequences such as speech or text, carrying information along step by step.
- Transformers use a mechanism called attention to relate parts of a sequence to one another, and are the basis of modern language models.
Why they became so prominent
Neural networks date back to the 1950s, but several things came together in the 2010s: very large datasets, graphics processors that can do huge numbers of calculations in parallel, and improved training techniques. Together these made large networks practical, and results on images, speech and text improved dramatically.
Limitations
They need data and computing power. Training large networks can require enormous resources, although small networks can be trained on ordinary computers.
They are hard to interpret. With millions of parameters, it is difficult to say exactly why a network gave a particular answer. Research on explainability aims to help.
They can overfit. A network may memorize its training examples and perform worse on new ones. Techniques such as testing on held-out data help detect this.
They inherit data problems. Biased or unrepresentative training data produces biased results.
They can be brittle. Inputs unlike those in training, or deliberately altered ones, can lead to confident mistakes.
They are not always the best tool. For small, structured datasets, simpler methods often do as well and are easier to explain.
Where to go from here
If you want to build intuition, try an interactive visualization that lets you watch a small network train, or write a very small network in code. Understanding the basics of machine learning first, particularly training data, testing and overfitting, makes everything else easier. Neural networks are impressive pattern-learners, but they are tools with known strengths and weaknesses, not minds.