Lamps
in the dark

Intuition over algebra

When I was learning hidden Markov models, the model itself was understandable to me, and so was the algebra behind it: how you actually arrive at the answer. But the picture never clicked. The geometry, the image of the thing, never formed in my head, so I had the steps without ever having an intuitive feel for what was going on. And that intuition is, to me, the part that matters most when you are trying to understand a concept.

The first time I had to build one of these for real was during my quant research internship at a crypto trading startup in London. I used a hidden Markov model to guess which "state" the market was in each hour: drifting quietly, going nowhere, or running hard in one direction. None of that is special to markets, though. The same machine reads speech, tags stretches of DNA, tracks where a robot thinks it is, and labels messy handwriting. Anywhere a hidden situation changes slowly and leaks clues as it goes, this is the tool people reach for.

A model like this has two moving parts. One says what each hidden state looks like: the kind of readings that state tends to produce. The other says how the states follow one another: whether calm tends to stay calm, whether a streak tends to keep going. The model begins knowing neither, and EM is how it learns both at once from nothing but the unlabeled readings.

Most courses hand you the update equations and trust the symbols to carry the meaning. They never did for me. What follows rebuilds the whole thing as a picture you can hold.

Every lamp lights the whole plane

Let me make the picture concrete before it gets abstract. Say a fresh reading arrives every hour, and each reading is a small bundle of numbers. Three, so we can draw it: maybe how fast things are moving, how spread out they are, and how much the recent past predicts the next step. (For example, in finance these three could be volatility, correlation, and a persistence measure like the Hurst exponent. The exact names do not matter here.) Stack those three numbers and you have a point. Drop the point onto a plane and the hour becomes a single dot. A week of hours is a loose cloud of dots floating in that space.

So imagine there are three different modes the world can be in, and let us call them lamps for now. What we are trying to do is sort the hours, every single dot we have, into these lamps. Each lamp sits at one spot, the most typical reading for its mood, and throws a coloured glow outward. There are three of them here, green, blue, and red, and the same three colours show up in every diagram below.

The tempting first picture is a map chopped into territories: this patch belongs to one mood, that patch to another, and a dot simply lands inside one of them, sort of like fences. Some of you might even think of it as k-means clustering, where every point is handed to its nearest centre. It is easy to draw. It is also wrong, and the wrongness turns out to be the entire mechanism, so it is worth changing the approach.

A mood does not own a patch. A mood is a lamp, brightest at its centre and fading as you move out (but never going fully dark anywhere; look at the diagram below and you can watch the glow thin toward the edges without ever stopping). Green's glow out at the far corner is faint. It is still not zero. Every lamp reaches every point on the plane, just with wildly different brightness.

So a dot is never simply "green and not red." It is lit by all three lamps at once, brightly by the near one and faintly by the far ones. The honest question is not which region it sits in but what mix of light it catches: this dot is 80% green light, 15% red, 5% blue. Those three shares always add up to 100%, and that mix, how much of each lamp's light a dot is catching, is the thing the whole algorithm is really about.

Drag the lamps below. The default view shows the truth: soft overlapping glows that blend where they meet, mixing into new colours wherever two beams cross. Turn a lamp's weight up and watch its territory actually grow and pull in dots near the edge, not just glow brighter in the same spot. Flip to fences to see the seductive lie, hard borders with every dot handed to one owner, and notice how brittle the seams are.

Now drop down to a single dot and ask what it actually sees. Three beams land on it at once, just at different strengths, and its mix is simply its share of each colour. The figure below freezes one dot near the green lamp and magnifies it: mostly green, with some red and a little blue. Around it you can watch the beams add like real light. Up top, where green and blue overlap, the colour shifts toward cyan. Down the middle, green and red pile into a warm yellow. In the centre all three meet and wash toward white. And out where one beam barely reaches, the other two split the dot between them.

How much of each lamp does a dot catch?

You might expect each lamp's glow to be a perfect circle. Look back at the diagrams, though, and the glows come out stretched, more like ovals than circles. That stretch is not decoration, and it changes what the word close even means. If you took a plain ruler and measured the raw centimetres from a dot to each lamp, you would sometimes get the wrong answer: a dot can sit nearer the green lamp in plain distance and still belong mostly to the red one. How can that be? Because a wide, sprawling mood reaches far while a tight, disciplined mood barely reaches past its own centre, so each lamp measures distance in its own stretched way. The fix is to stop using a plain ruler and use one shaped to fit each lamp.

Picture a lamp's spread as two arrows planted at its centre, at right angles. A long arrow down the direction the readings stretch most, and a short arrow across the direction they barely vary. A round lamp has two equal arrows. An oval one, think of a surfboard, has one long and one short. Now stop asking "how many centimetres away is the dot" and ask instead, "how many of this lamp's own arrows out is it, along each arrow?"

A step along the long arrow barely uses up the ruler, so far still feels near. A step along the short arrow burns through the ruler fast, so a little way out already feels distant. That makes the short direction the sensitive one, the direction where a dot most quickly stops looking like it belongs. Turn that stretched distance into a brightness with a smooth bell-shaped fall-off (close means bright, far means dim, never quite zero), do it for all three lamps, and the dot now has three raw brightnesses.

One last move makes them the percentages: divide each lamp's brightness by the total of all three. That is why the mix always lands on something like 80 / 15 / 5 that sums to 100. You are asking what share of this dot's total light comes from each lamp. Computing that share for every dot, with the lamps held where they currently sit, is the whole E-step. The E is for expectation, because each share is the expected amount that the dot belongs to that mood.

Two honest notes while we are here. The oval shape is an assumption, not a fact about the world. We are choosing to model each mood as a smooth elliptical blob because it is easy to compute with, and that choice is weakest out in the tails, which is exactly where the interesting surprises live. And the brightness is not distance alone: it is also scaled by how common the mood is, so a rare lamp needs strong local evidence before it can claim a dot. That second factor is the weight slider you were dragging a moment ago.

If that looked like PCA, you were right

Some of you will have felt a bump of recognition at "long arrow, short arrow." Those two arrows are a principal component analysis of that lamp's cloud, the same object under a different name. The long arrow points along the direction of most variation. Project the cloud onto it and it spreads the most. The short arrow points along the least variation. The thing that holds the stretch and the tilt, the covariance matrix, is exactly what PCA reads its axes off of.

The arrows are forced to be at right angles, by the way, and not by convention. A covariance matrix is symmetric, and symmetric matrices always hand back perpendicular axes. One nice twist: ordinary PCA usually throws the short arrow away to compress data, while the lamp leans on the short arrow hardest, because that is the direction where a dot most quickly stops looking like one of its own. Same arrow, opposite attitude toward it.

Centre first, then the arrows

The E-step coloured every dot with the lamps held still. The M-step does the opposite: freeze the colours, move the lamps. Each lamp slides to the weighted middle of the dots, where every dot tugs the lamp toward itself in proportion to how much of that lamp's colour it caught. A dot that is 80% green yanks the green lamp hard. A 5%-green dot barely nudges it. Then, around the new centre, you redraw the two arrows from the spread of the dots that softly belong to it. The M is for maximization, because that move is the one that makes the dots you actually observed least surprising under the updated lamps.

This whole move resembles k-means clustering, except we never draw a hard border. Every dot still receives all three lights, just in different proportions, so it tugs all three lamps at once instead of being handed to a single nearest one. That soft pull is the only difference, and it is the difference that makes everything later work.

This is also why a bad centre wrecks the shape. Measure spread from an off-balance point and you do not just inflate the arrows, you rotate the long one off true, and the oval becomes a distorted account of the cloud. One small guardrail lives here too: a lamp can collapse onto a single dot if you let it, its spread shrinking toward nothing and its brightness spiking toward infinity. Real code quietly forbids that by refusing to let a lamp shrink below some minimum size. Otherwise "tightening onto the real shape" can degrade into a lamp impaling one lucky point.

Colour, move, colour, move

Now alternate. Colour the dots with the lamps frozen. Move the lamps with the colours frozen. Recolour. Re-move. That back and forth is the EM algorithm: expectation, then maximization, repeated until the lamps stop sliding. Watch it run below, starting from three random lamps dropped on a cloud they do not fit, then walking into place pass by pass.

A caveat that the tidy animation hides. EM only ever promises that a pass will not make the fit worse. It climbs to the nearest good-enough arrangement (a local optimum, what optimization people call a local minimum), not the best one that exists. Start the lamps somewhere else and they can settle into a different, worse layout and sit there happily, because "nothing is moving" only means you found a resting spot, not the right one. In practice you run it a few times from random starts and keep the best. The clip above always tidies up. Real data sometimes parks a lamp in a ditch and calls it done.

The Light Keeper, the second force

There is a second force I have been holding back, and it is the thing that truly separates this from the k-means clustering it keeps resembling. Picture a Light Keeper, someone who walks the dots in order and checks that none of them is doing anything strange. As he walks he keeps a notebook. Dots bathed heavily in green light tend to stay green for a while; it almost never goes green, green, green, then a hard jump to mostly red, then green again. He has simply never seen this little world flip that fast.

Now watch the picture below. He walks the green trail hour by hour, x_n, then x_n+1, then x_n+2, all of them deep green. Then he reaches a stray dot that, by raw distance, looks far from the green light and almost sitting on the red lamp. Down in the corner the red lamp notices it and starts pulling, dragging its own centre toward this single dot. The Light Keeper says no. The hours on both sides of the stray are deep green, and green, then a hard red, then green again would be the most surprising thing he has seen all day. So he writes the stray down as green after all, left sitting just on the edge of the green region, and the red lamp lets go.

The reason he can overrule the geometry is that the dots are a sequence, hour after hour, not a scatter thrown down at random. That is why they are labelled x_n, x_n+1, and so on. Real processes carry this kind of inertia. A river running down a mountain does not flow, dry up completely, and come roaring back inside three hours. Moods change with some normality, and the model leans on exactly that.

So the Light Keeper does quiet, important work. He makes sure no single stray drags a lamp out of place, and that every dot ends up where the rhythm of the trail says it belongs. The catch is timing. If you let him do his accounting only at the very end, he never gets to vote while the lamps are still moving; he just describes where they drifted once it is too late to hold them. Run him every pass, and his veto becomes a live force that keeps a lamp from chasing one lonely dot, over and over, until the shapes settle on the real clusters.

The points arrive in order

Until now the dots were a loose cloud. But they arrive in sequence, hour 1, hour 2, hour 3, and the light each one catches is sticky: green tends to follow green, a streak tends to keep going. That stickiness, the rule the Light Keeper reaches for when he decides where a dot belongs, is the transition matrix: three moods across, three down, each cell holding how strongly one mood tends to be followed by another. Think of it as the notebook he keeps of how this little lamp world behaves, read straight off the trail in the same soft currency as everything else.

So a dot's mix now comes from two things at once: how brightly each lamp hits it, and how plausibly the surrounding trail allows that mood. A dot sitting right under the green lamp can still come out mostly blue if calling it green would force the trail to flicker blue, green, blue across three hours, a move the notebook says almost never happens.

With time in the picture the names settle into place. The whole colour-move-colour-move loop, now including the grid, is EM. Applied to this lamps-plus-trail model it earns its own name, Baum-Welch. The two-pass trick inside the colouring step is forward-backward. In a typical codebase, the single call fit() is the whole Baum-Welch loop with forward-backward humming inside it.

Once the lamps are fixed, thread the needle

Fitting is done: the transition matrix, the lamp centres, and the beam shapes are all fixed now. A different job comes next. Given a fresh sequence of hours the model has never seen before, which single sequence of moods is the system most likely to have walked through? Not the favourite mood at each hour on its own, but the best whole path, one mood per hour, chosen so the entire chain stays consistent with the stickiness grid. That is the Viterbi algorithm.

Here is the idea in plain steps. Every dot already carries its mix of light, say 80% red, 15% blue, 5% green. The first hour arrives mostly green, so we call the mood green. The next five hours are mostly green too, so we stay green. Then the sixth hour lands mostly red, more red territory than anything else, so we switch to red. Walking the trail like this, reading each hour's mood from the light it caught while respecting how willing the grid is to switch, is what Viterbi does. It is a close cousin of dynamic programming: each step reuses the best answer from the step before instead of rebuilding the whole path from scratch.

Picture a trellis: a column of three mood-nodes for each hour, time running left to right. Every node carries a score for "the best story that ends here," built by extending the best story from the previous column through the grid and the lamp brightness at this hour. Sweep left to right filling in scores, then walk back right to left following each node's best predecessor, and one brightest thread lights up. Step it below.

One caveat I glossed: the single most likely path is not always what you want, because committing to one mood per hour quietly re-imposes the hard borders. Sometimes the full mix matters: a dot that is 60% red is also 20% blue and 20% green, and keeping that whole mix (marginal, or posterior, decoding) is the right call when the small probabilities are the point, as in a risk measure that cares about the tail. Same data, different question.

The model hands you nameless lamps

Here is the part that surprises people. The model never names its moods. fit() hands back three lamps with no labels, lamp 0, lamp 1, lamp 2, in an order that can even reshuffle between runs. It found three blobs and three handoff tendencies. It has no idea that one of them is "calm" and another is "wild." Attaching meaning is your job, and you do it by looking at where each lamp's centre sits.

How you name them depends entirely on what your three numbers measured. If one of those numbers separates the moods cleanly, sort the lamps along it and read the order straight off. That is the whole step. The sort also quietly fixes the reshuffling problem: whichever index the model handed you, the name you care about always lands on whichever lamp is currently most extreme along your chosen number.

There are many ways to pick that number, and the right one is whatever makes the moods mean something in your problem. In the markets case I mentioned, people lean on a single number called the Hurst exponent, which sorts a path from "snaps back" to "runs away." It is a nice example because it turns three anonymous blobs into three words you can act on, and it is exactly the kind of thing that needs a little finance vocabulary, so I have tucked it out of the way.

Open the Hurst exponent (with a slider to play with)

Once the lamps have names, you can finally ask things in plain language: which mood was the system in at 3pm, how often does it switch, what usually comes after a long calm stretch. The shapes came from the math. The meaning comes from this one human step of looking and labelling.

Every picture, its textbook name

A cheat sheet for the names, for when you take this to a paper or a library. The left column is what I have been calling things, the middle is what you will see them called out there, and the right is a one-line reminder of the job each one does.