A History of AI, Told Through One Number

01The one thing

You already do this. You just don't call it weighting. Meeting someone for the first time, you decide how much to trust them by leaning hard on a couple of signals (whether they actually listen, how they treat the waiter) and ignoring others completely (what car they drive). Nobody handed you that ranking. You built it over years of misjudging people and adjusting.

That ranking is the whole idea of a weight: how much a given signal counts toward the answer. An artificial neuron is the same move in numbers. Each signal is multiplied by its weight (high, low, or zero), the results are added up, and if the total clears a line, the neuron fires.

Stack billions and you have a language model. The behavior has no second ingredient. No rules, no logic anyone wrote for what it should say. Just numbers saying how much things matter to each other, found by training rather than typed in by hand.

Which is what makes the question of who sets them the only one that's ever mattered. Seventy years, two collapses, one long feud, and the thing on your phone. All of it is that.

021943, a neuron becomes arithmetic

Warren McCulloch, a neurophysiologist, and Walter Pitts, a logician who taught himself the field as a teenager and was more or less homeless when the two of them met, published a paper describing brain cells as simple logical switches. Inputs, a threshold, an output. On or off.

They meant it as biology. What it proved by accident is that you can build thought-shaped things out of arithmetic.

It couldn't learn, though. A person set the weights by hand, and that person had to already know the answer. Which makes it a description of a brain, not a thing that behaves like one.

That gap took fifteen years to close.

031958, and the worst press release in science

Frank Rosenblatt was a psychologist at Cornell, and the piece he added is almost insultingly simple.

Show it an example. Let it guess. If the guess is wrong, nudge the weights slightly toward what would have been right. Do it again.

That's the perceptron learning rule, and it is the ancestor of everything running today. Nobody tells the machine what matters. It finds out by being wrong at scale.

He first ran it as software on an IBM 704, and the Navy held a press conference in 1958 to show it off. The New York Times covered it under a headline about a device that learns by doing, describing a machine expected to eventually "walk, talk, see, write, reproduce itself and be conscious of its existence." The New Yorker called it the first serious rival to the human brain ever devised.

What the machine had actually done that day was learn to tell left from right, after about fifty tries.

The hardware version came around 1960. The Mark I Perceptron was room-sized, with a 20x20 grid of photocells for an eye, wired to banks of motor-driven potentiometers that served as the adjustable weights. Which means the weights were knobs, and learning was a set of small motors turning them. You could stand in the room and watch a machine physically change its mind.

Every hype cycle in AI since has the shape of that press conference. A real breakthrough, correctly spotted as a big deal, then described in language that runs ahead of what the thing can do yet. Worth holding onto when you read anything about AI now, including this.

04The problem a straight line can't solve

Take a small logic puzzle called XOR. Two inputs, each either on or off. You want a yes when they differ (one on, one off) and a no when they match. It's the logic of a hallway light wired to a switch at both ends: flip either switch and the light flips. Sounds like nothing.

One layer of weights cannot do it. Not "hasn't managed yet." Cannot, provably.

The reason is geometric. A single layer can only draw a straight line through your data, one side yes and the other no. XOR needs a division a straight line can't produce. And once you know to look for that shape, you find it in most problems worth solving.

The fix was visible even then. Stack layers and the network can bend the line. The catch was that nobody could train a stack. The learning rule works on one layer because when the answer comes out wrong you know precisely which weights to blame. Add layers and the blame goes murky. Which weight, in which layer, caused this?

So the field had a machine that could learn but not handle anything real, and a design that could handle real things but not learn.

05The 1969 story everyone repeats wrong

Marvin Minsky and Seymour Papert published a book called Perceptrons laying out exactly that limit, and the standard telling is that it single-handedly killed neural networks and triggered the first AI winter. One book, funding gone, field dead.

That version is in most explainers and it doesn't survive contact with the sources.

Minsky and Papert had been arguing against this approach since around 1965, at conferences and in circulated drafts. By 1969 most researchers had already drifted away, worn down by lack of progress, and the rules-based camp had largely won the funding fight before the book existed. It landed in a room that was already emptying and picked up its reputation as the assassin partly through timing.

Rosenblatt died in a boating accident in Chesapeake Bay in July 1971, on his 43rd birthday. When the book was reissued in 1987, it carried a dedication to him.

Worth knowing which version you're repeating, because the tidy one makes it sound like ideas die from criticism. They mostly die from people quietly leaving.

06Thirty years of typing rules in by hand

Left out of most tellings: for about three decades the mainstream of AI wasn't weights at all.

The other approach was rules. Write down what you know as explicit logic and let the machine reason over it. If fever and cough, consider flu. Knowledge as statements a human typed in.

This was the sensible bet, not the dumb one. Rules are readable. You can debug them. You can ask the system why it decided something and get a real answer, which is the exact thing everyone now complains AI can't do.

It produced expert systems that worked in narrow slices (diagnosing infections, configuring hardware orders), and a genuine industry grew around them in the 80s.

Then it hit the wall it was always going to hit, which is that a human has to type in every single thing. Every rule, every exception, every exception to the exception. Systems grew past a few thousand rules and started contradicting themselves in ways nobody could untangle. And the parts we do without thinking, seeing, hearing, reading a room, turn out to be the hardest to write down, because nobody can say what rules they're following.

The market collapsed in the late 80s. Second winter.

Both approaches failed, which is the part that gets dropped. Rules failed because humans can't articulate what they know. Weights failed because nobody could train more than one layer. Same era, different walls.

071986

The unlock is backpropagation, from a 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams. Earlier versions existed (Linnainmaa in 1970, Werbos in 1974), but this is the one that landed.

It solves the blame problem. Work the error backwards through the network, layer by layer, calculating how much each weight contributed to the mistake, then nudge all of them slightly toward better. Millions of examples later, a stack is trained.

Still the engine today. Bigger, faster, same idea.

And look at what it actually is: guess, measure the error, nudge the weights. Rosenblatt's rule with the layer problem solved. 1958, unstuck.

Hinton shared the Nobel Prize in Physics in 2024, partly for this. Thirty-eight years is a long time to wait for a phone call.

08Why a finished idea sat there until 2012

Backprop worked in 1986. Deep learning didn't take over until 2012, and that 26-year gap is the most useful thing in this history.

It wasn't ideas. The math was done. Yann LeCun had these networks reading handwritten digits on real bank checks in the 90s. It worked. It just couldn't be pushed further.

Two things were missing and both are boring.

Data. These systems need an absurd number of examples, and before the internet, assembling a large labeled dataset meant paying people to sit there labeling. ImageNet landed in 2009, millions of labeled images, built through a lot of unglamorous coordination nobody writes hero stories about.

Compute. Training is billions of small multiplications, which happens to be exactly what a graphics card does to render video games. Nobody designed GPUs for this. Researchers noticed the hardware fit.

In 2012 a network called AlexNet, from Alex Krizhevsky, Ilya Sutskever, and Hinton, entered the ImageNet competition running on gaming GPUs and posted a top-5 error rate of 15.3 percent, meaning the correct label sat in its top five guesses about 85 percent of the time. Second place was more than ten percentage points behind. In a benchmark where people fought over fractions, that ended the argument.

Nothing conceptual had changed since 1986. The idea was finally allowed to run.

The idea was finished in 1986 and useless until 2012. What showed up wasn't insight. It was cheap data and gaming hardware, arriving from two directions nobody in AI controlled.

09Weights that point at each other

Last piece. The transformer, 2017, from a Google paper whose title gives away the ending: "Attention Is All You Need."

The problem it fixed: earlier systems read text one word at a time in order, holding a running summary, and that summary gets overloaded. Things far apart in a sentence lose track of each other.

Attention lets every word look at every other word directly and decide how much each one matters to it. In "the trophy didn't fit in the suitcase because it was too big," the word "it" checks both nouns and weighs them.

Notice what attention is, though. Weights again. Computed fresh for whatever you typed instead of fixed after training.

Same 1943 neuron, same 1958 rule, same 1986 blame math, rearranged so the weights can point at each other. That's the entire architecture under everything you've used.

10The bitter part

In 2019 Rich Sutton wrote a short piece called The Bitter Lesson, and it's the closest thing here to a moral.

His claim: for seventy years the same thing keeps happening. Researchers carefully build in human knowledge about a problem. Chess strategy, grammar rules, which features matter in an image. It works, and it feels like science. Then someone throws a dumber, more general method at more compute and beats it, and all that encoded expertise turns out to be dead weight.

Chess, Go, speech, vision, translation. Same result every time.

He calls it bitter because it flatters nobody who does the work.

I don't buy the strong version. The transformer was human insight. Attention was a design choice a person made, and "just add compute" reads far cleaner in hindsight than it did to anyone deciding what to add compute to. What holds up is the narrower claim, which is that betting against scale has a perfect losing record, and the people who lost that bet were usually the ones most certain their expertise was the important part.

11So nobody can read it

Back to where this started.

We understand training completely. The recipe is public and taught in undergrad courses. Guess, measure the error, nudge every weight, repeat trillions of times.

We invented the search. We did not pick the result.

Nobody chooses the weights. Nobody has inspected them. Hundreds of billions of numbers, each adjusted by microscopic amounts an unfathomable number of times, and what comes out is the residue of a process no human supervised at any point.

They aren't hidden, either, which is the next assumption people make. You can download an open model right now and print every number. They look like 0.0231, -1.4432, 0.8871. And they tell you nothing, because meaning isn't stored in any single one. Concepts get smeared across many weights at once, and each weight takes part in many different concepts depending on what else is happening around it. There's no dictionary. There was never anyone around to write one.

So we have an artifact we built and now have to study experimentally, the way you'd study something you dug up. There's a field for this called interpretability, and it is mostly unsolved.

The short version: we automated the writing of the program, and the price was being able to read it.

That's not a bug that crept in late. It's the same trade Rosenblatt made in 1958 when he chose a machine that finds its own weights over a human writing them down. The rules camp kept readability and lost, because humans can't type in everything they know. The weights camp took the problems nobody could write down and gave up ever knowing what the solution says.

Nobody framed it as a trade at the time. It turned out to be one.

Three things that fall out of this if you use these tools for work:

There's no rulebook inside, so stop hunting for the line to fix. When a model gets something wrong, nobody can open it and point. That's why prompting is trial and error instead of configuration.

A confident answer isn't automatically a correct one. Good answers and wrong ones come out of the same process, so it's worth checking the ones that matter instead of going off the tone.

When capability jumps, the boring explanation, more data and more compute, is usually most of it.

12The frozen machine

One last thing, because it's where people's intuition breaks.

Paste five examples of some invented task into a prompt, something the model has never seen, and it will do the sixth one correctly. It picked up a new task on the spot. Nothing inside changed.

That looks like learning, and it isn't, in the sense that matters. Water through pipes. The pipes were shaped once, during training, and never change again. Water flows, takes on a pattern, the pattern does something useful, and when the flow stops nothing remains. Close the chat and it's the identical machine it was before you typed a word.

When you learned to do your job it changed you, and you'll still know it next week. The model can't do that mid-conversation. Every session starts from the same frozen thing.

Which is the whole history in one behavior. A machine that appears to learn, built by a process that already finished, made of numbers nobody selected, doing something nobody can fully explain.

The explaining part is still open. Seventy years in, that's the piece still sitting there.

Sources

The originals, if you want them instead of my read: McCulloch and Pitts (1943); Rosenblatt's 1958 perceptron work and the New York Times piece from July 8, 1958; Minsky and Papert's Perceptrons (1969), plus Yuxi Liu's essay on the perceptron controversy for the corrected history in the 1969 section; Rumelhart, Hinton and Williams (1986); Krizhevsky, Sutskever and Hinton (2012) for AlexNet; Vaswani et al. (2017); Rich Sutton's The Bitter Lesson (2019). Anything wrong is mine, not theirs.