Neural Networks: Backpropagation and the Long Climb Back

The hidden layer insight: if one layer can only draw straight lines, multiple layers can warp the space itself.

The Ideas in Conflict

The AI Winter didn’t freeze everyone equally. By the early 1980s, expert systems were the dominant paradigm — commercially deployed, DARPA-funded, institutionally respectable. The approach: hire a domain expert, interview them exhaustively, encode their knowledge as explicit IF-THEN rules, run an inference engine over them. XCON, deployed at Digital Equipment Corporation in 1982, was saving the company $40M a year.¹

The competing ideas were:

• Connectionism — the underground view that intelligence emerges from learning, not from hand-crafted rules. The PDP (Parallel Distributed Processing) group at UCSD — Rumelhart,David Rumelhart, first author of the 1986 backprop paper, developed Pick’s disease (frontotemporal dementia) in the mid-1990s and retired in 1998 unable to work. He died in 2011, unable to speak or write for his last decade. The Rumelhart Prize — the field’s highest honour, named after him — was first awarded in 2001 to Hinton. Rumelhart was already incapacitated. He died one year before AlexNet. McClelland, HintonHinton moved to Toronto in 1987 partly on principle: he refused DARPA funding with defence applications. He stayed 25 years. His students — LeCun, Sutskever, Krizhevsky — built the modern field. The principled retreat that looked like obscurity was the founding of an intellectual dynasty. — were its intellectual centre. Their 1986 two-volume manifesto Parallel Distributed Processing was a declaration of war.
• Statistics — a separate discipline entirely. Statisticians had logistic regression, linear discriminant analysis, and maximum likelihood. They considered neural networks unprincipled black boxes with no theoretical guarantees.
• Control theory — Widrow and Hoff’s ADALINE (1960) had used gradient descent on single-layer networks. The mathematics was there; nobody had extended it to multiple layers.

The question that separated these camps: does intelligence come from knowledge representation, or from learning? Expert systems said: encode what you know. Connectionists said: learn it from data.

The expert systems camp had funding, prestige, and commercial success. The connectionists had a better idea.

1986 — Backpropagation: The Chain Rule as Credit Assignment

The problem Minsky left open in 1969 was credit assignment: given an error at the output, how do you apportion blame to hidden units deep in the network? A hidden unit’s error isn’t directly observable — you only see the final output. How does the signal get back?

The Ideas Rosenblatt Was Working Against (Redux)

The block wasn’t mathematical difficulty. The chain rule is 200 years old. The block was conceptual:

• Nobody believed multi-layer networks were worth training. Minsky’s verdict had poisoned the well.
• Credit assignment seemed to require global coordination — some central controller surveying the whole network, deciding who was responsible. Local units computing their own blame seemed circular.

The insight that broke the impasse: the architecture is a composition of functions. Each layer applies a function to the previous layer’s output. The derivative of a composition is — by the chain rule — a product of local derivatives. Each unit only needs to know its own local gradient and the gradient flowing in from the layer above.

No global coordination required. The forward pass computes activations layer by layer; the backward pass computes gradients layer by layer, in reverse. The error flows backward through the same structure the activations flowed forward through.

This wasn’t a trick. It was the only tool that correctly decomposes the gradient given the compositional structure. The architecture already contained the solution. The same structure that made credit assignment seem impossible made it uniquely tractable.² What led Werbos to backprop in 1974 was Freud, not control theory — he was trying to mathematise Freud’s idea of backward credit flow through a psychic system. His Harvard committee called it “crazy, megalomaniac, nutzoid.” He survived by secretly embedding the algorithm inside a political science thesis, eating soybean soup in a Roxbury slum to conserve money. He couldn’t publish it cleanly for eight more years. Sources: thesis, 1993 interview.

The Computational Graph

Backprop is most clearly understood on a computational graph — a directed acyclic graph where each node is an operation and edges carry values (forward) or gradients (backward).

Gradient Descent on a Loss Surface

Training is finding the lowest valley in a loss landscape. Gradient descent says: measure the local slope, take a small step downhill, repeat.

Non-Linear Activations: Why Linear Layers Are Useless Stacked

If you stack linear layers: $W_2(W_1 x) = (W_2 W_1)x$ — still one big matrix. No matter how many layers, you are still cutting space with a single straight line. Depth without non-linearity collapses.

Non-linearity introduces kinks that let the network learn curved, complex boundaries. Each activation breaks the linearity at that layer, so subsequent layers compose genuinely different transformations.

The competing ideas about which non-linearity to use:

• Sigmoid (1986) — biologically motivated, smooth, bounded. Problem: derivative peaks at 0.25. This mattered enormously once networks went deep.
• Tanh (zero-centred sigmoid) — a small improvement. Same peak derivative problem.
• ReLU — Rectified Linear Unit, $\max(0, x)$. Known in neuroscience for decades (biological neurons are roughly ReLU-shaped). Nobody thought to use it in artificial networks until Glorot & Bengio (2010) analysed why sigmoid was failing, and Nair & Hinton (2010) showed ReLU worked better in practice.³

The Vanishing Gradient: Why Depth Still Failed

Backprop solved multi-layer training. It did not solve deep networks. The field discovered this the hard way through the late 1980s and 1990s.

The Problem

Sigmoid’s derivative peaks at 0.25. Chain-multiply through 10 layers:

Early layers receive nearly zero gradient. They don’t learn. The network is theoretically deep but effectively shallow — only the last few layers are actually training. The further back you go, the less information arrives.Hochreiter proved this rigorously in his 1991 diploma thesis at TU Munich: for sigmoid activations, the gradient-scaling product along any path is provably less than 1, decaying exponentially with depth. Increasing weights makes it worse — as weights grow, f’ shrinks toward zero. The thesis was written in German, by an unknown student, and received almost no attention.

The Fixes

The fixes arrived over two decades from three separate communities:

None of these arrived together. ReLU required recognising sigmoid as the culprit. BatchNorm came from a Google team noticing training became unstable as depth increased. ResNets arose from an anomaly: adding more layers was making performance worse, not better — which shouldn’t happen if the extra layers could simply learn identity mappings.

The solution to vanishing gradients for recurrent networks came earlier and from a different direction. Hochreiter and Schmidhuber’s LSTM (1997) introduced the constant error carousel — a memory cell with an identity self-connection (weight 1.0), so the gradient neither grows nor shrinks as it circulates through time.Hochreiter named it the “constant error carousel” because error signals could circulate indefinitely without decay. Gating units decide what enters and leaves the cell. The 1997 paper received ~15 citations/year for five years. It now has ~98,000 — one of the most cited papers in ML history.

Universal Approximation

A neural network with one hidden layer and enough neurons, using a non-linear activation, can approximate any continuous function to arbitrary precision.Proved independently by Cybenko (1989) for sigmoid activations and Hornik (1991) for general activations.

The proof is constructive: each neuron carves out a “bump” in the input space. Stack enough bumps, you can approximate any shape.

But notice what Universal Approximation does not guarantee:

• How large the network needs to be (could be exponentially large)
• That you can learn the right weights (existence ≠ findability)
• That the network will generalise to new data

These three gaps — efficiency, learnability, generalisation — define the open problems of the next 30 years.

1989–1998 — LeCun and CNNs

The vanishing gradient problem stalled deep fully-connected networks. But Yann LeCun found a different path: exploit the structure of the input domain.

The competing approaches to vision in the late 1980s:

• Hand-crafted features Computer vision researchers spent years designing features based on their understanding of image structure — edge detectors, HOG descriptors, SIFT. These features were engineered and then fed to a classifier. Learning was not part of the pipeline.

• Fully-connected networks — treat each pixel independently. An FC network must learn that a cat’s ear looks the same whether it appears top-left or centre-right, redundantly, for every position.

LeCun’s insight: natural images have spatial structure. Patterns that matter (edges, textures, shapes) appear at multiple locations. The same detector should work everywhere — share weights across positions.

The Insight

A convolution is a filter that scans across the input, applying the same small set of weights at every position. One filter learns one kind of pattern (an edge, a curve, a texture) and finds it wherever it appears.

This buys two things:

1. Translation equivariance — if the pattern moves, the activation moves with it. The network doesn’t have to re-learn “cat ear in top-left” separately from “cat ear in centre.”
2. Parameter efficiency — instead of a weight connecting every input pixel to every hidden unit ($784 \times 500 = 392{,}000$ parameters for a tiny MNIST network), a $5 \times 5$ filter has 25 parameters and applies them everywhere.

Stack convolution layers and you get a hierarchy: the first layer learns edges, the second learns textures built from edges, the third learns shapes built from textures. Each layer abstracts over the layer before. LeCun arrived at Bell Labs in 1988 after a postdoc under Hinton in Toronto. His account of weight-sharing is undramatic: “the logical thing to do.” The 1989 paper explicitly cites Hubel and Wiesel’s 1962 discovery of oriented edge detectors in cat visual cortex. Those cells were hand-wired by evolution; LeCun’s could be learned end-to-end with backprop. That combination had not been made before.

What LeNet Actually Did

LeCun’s LeNet (1989, refined to LeNet-5 in 1998) was not a research curiosity. It was deployed in real systems: by the late 1990s, LeNet was reading handwritten cheques at ATMs across the United States, processing an estimated 10–20% of all cheques written in the country.⁵ Deep learning was running production systems while the academic field was still debating whether neural networks worked at all.

The architecture:

Input (32×32) → Conv → Pool → Conv → Pool → FC → FC → Output (10 classes)

Each conv-pool block reduces spatial size while increasing depth. The FC layers at the end combine learned features to produce a classification.

What It Left Open

CNNs worked because images have spatial structure. The same principle — exploit domain structure through architectural inductive bias — would take decades to apply elsewhere:

• Text has sequential structure → RNNs, then Transformers
• Graphs have relational structure → GNNs
• Physics has symmetry structure → equivariant networks

The CNN insight is really a meta-insight: the right inductive bias, baked into the architecture, can substitute for enormous amounts of data and compute. This remained the dominant design principle until scale made it optional.

What This Era Left Open

The backprop era solved the training problem for shallow networks and, with CNNs, for structured domains. What remained:

• Scale: LeNet worked on small images (32×32). Real-world images are orders of magnitude larger. Training on them required compute that didn’t exist yet.
• Data: Supervised learning requires labels. Getting millions of labelled examples requires infrastructure that also didn’t exist yet.
• Depth for unstructured domains: Fully-connected deep networks still failed on raw pixels at scale. The vanishing gradient fixes (ReLU, BatchNorm, ResNets) arrived in 2010–2015 — not yet.
• Generalisation theory: Why do these networks generalise to unseen data? Especially: why do overparameterised networks generalise? This question remains genuinely open.

The door was briefly reopened in 2006 by Hinton’s deep belief network paper,Hinton’s insight was complementary priors: if the prior over a hidden layer matches the posterior the layer below computes, the “explaining away” problem cancels exactly. This made greedy layer-wise training of RBMs theoretically justified. Pretraining turned out to be unnecessary once ReLU arrived — but it bought six years of renewed research and ended the second neural network winter. which showed that greedy layer-wise pretraining could give gradient descent a starting point good enough to train deep networks. The compute and data problems were solved together in 2012 — the story of AlexNet, ImageNet, and the GPU revolution.After AlexNet’s 2012 victory — 15.3% top-5 error vs second place’s 26.2% — Hinton and students Krizhevsky and Sutskever formed DNNresearch and ran an auction. Hinton’s back condition prevents flying; bidding happened by laptop between conference sessions in Lake Tahoe. Google, Microsoft, DeepMind, and Baidu bid. Google won for ~$44M. Hinton had spent 25 years working against the current. That is the next era.

Continued in Neural Networks: The Deep Learning Revolution