The odds are a choice

IThe number you inherit

EveryHistoryv1.212 OctSection III: the hit rate is now split by who controls the outcome.v1.16 OctAdded section IV, “Where it failed”, and a fifth line to the method.v1.03 OctPublished. plan comes with a number already attached. Most new companies close within a few years. Most people who start running in January have stopped by March. Most side projects never leave the drawer. I used to read those numbers the way you read a forecast: something that happens to you, and that you dress for.

A forecast does not care what you do next. Odds do. A base rate is an average over people who made choices, and most of them made the choices you are about to make. Take the number as given and you are quietly agreeing to make those choices too.

The trouble is that the number feels like news about the future, when it is mostly news about the past: what other people did, with what they knew and the habits they had. It tells you how the story usually goes. It says nothing about whether you have to tell it the usual way.

So I stopped asking what my odds were. I started asking which odds I was signing up for, and what it would cost to sign up for better ones. That is the whole idea. The rest of this essay is the method, and the part where it broke.

IIChoosing the class

Forecasters call it the reference class¹1Kahneman and Lovallo called the move “taking the outside view”. Tetlock’s Superforecasting is the readable version.: the set of past cases you decide your case belongs to. Pick the class and the number follows. People who start a company give you one figure. People who start one they have already run on the side for a year give you another. People who sold to ten customers before they quit anything give you a third.

None of these is the true number. Each is a door. The facts tell you which doors exist; they do not tell you which one to walk through. That part is a decision, and for most of my life I let it be made for me, usually by whichever statistic I had read most recently.

It helps to say the classes out loud. People who decide to write every morning have a miserable base rate. People who write at the same table, before opening anything, and stop mid-sentence so tomorrow starts easily, do much better. Same person, same goal, different door. The only thing that changed is which group I agreed to imitate.

The method fits on an index card²2The card is the first thing in my god.md, just under rule I, Forever Forward.. I keep it where I can see it:

1. 1Write the outcome I want, with a date.
2. 2Find three classes I could honestly belong to, and the base rate of each.
3. 3Pick the one I am willing to behave like. List what its members did that the others did not.
4. 4Do those things. Every Monday, check that I still belong to the class I picked.

The fourth line is the only hard one. Picking a flattering class takes a minute. Staying in it takes the rest of the year, and the class does not care about your reasons for leaving.

Finding the base rates is the dull part, and I do it roughly on purpose. A rough number from a decent source beats a precise one I will never look up. Most of the value is in writing three of them side by side. Seen together, they stop looking like fate and start looking like a menu.

IIINinety percent

For a year I wrote a confidence next to every decision I logged³3One row per call: the claim, the date, the confidence, the result. Forty-one rows so far this year.. Not a mood: a number, written down before the result came in. When I wrote ninety percent, I meant that out of ten things like it, nine would happen.

v1.2Changed in v1.2This paragraph used to give one hit rate for the whole year. A reader asked how it split. The split turned out to be the point.They did not. Of everything I marked ninety percent, about seven in ten came true, and the miss was not spread evenly. On outcomes I controlled, like shipping a build, finishing a run or sending the thing on the day I said, I was close to calibrated⁴4Calibrated: of all the things you call 90% likely, about 90% happen. You can be calibrated and still be wrong often. You just know how often.. On outcomes that ran through other people, I was badly overconfident. My ninety percent only meant ninety percent where the odds were mine to choose.

That changed how I use the number. A confidence is not a forecast about the world. It is a measure of how much of the outcome I have chosen to hold. When I write sixty percent next to something I care about, the useful question is not how to feel more sure. It is which part of this is still somebody else’s odds, and whether I can take it back.

Usually I can take back more than I think. A meeting I was waiting for becomes a document I send first. A launch that hung on one review becomes three smaller launches that hang on nobody. The number goes up not because I believe harder, but because I moved the outcome into a class where people like me behave differently.

There is a cost. Choosing a class means giving up the comfort of the average. If I fail inside a class I picked, I cannot say the odds were against me; I signed for them. I think it is a good trade. I would rather own a bad number than borrow a good one.

The log showed one more thing. My highest numbers were never on the things I had planned hardest. They were on the things I had already done once, the same way, with the same people. Repetition, not conviction, was what earned a ninety.

IVWhere it failed

The method has a failure mode, and I walked straight into it. Once you believe the odds are a choice, every bad outcome looks like a choice you made badly. In the spring I picked a class for an agent I was building: people who ship a working version every week and use it themselves every day. I stayed in that class for eleven weeks. I did every line on the card. It still did not work.

The class was right. The ground under it had moved. In week three, the service the agent ran on changed how it worked⁵5Not a complaint. The change was announced; I was too busy shipping to read it., and nothing on my card told me to look up from the work and notice. I had confused choosing my behaviour with choosing the world.

I could have caught it. The signs were there by week four: slower answers, more mornings spent fixing the agent than using it. But my card only asked whether I was still behaving like the class I had chosen, and I was. A method that only checks your behaviour will tell you that you are doing everything right, all the way to the end.

So the card has a fifth line now⁶6Added in v1.1. Most issues on the first version said the same thing: this is survivorship bias with a method attached. Fair., written under the other four:

5. 5Name the odds you did not choose.

Some of the number is still inherited. Tools change, bodies get sick, other people decide. Pretending otherwise is not courage; it is a more flattering way of being wrong. Naming those odds out loud is what keeps the rest of the method honest.

I still choose. I still check every Monday. But I write two numbers now: the odds I took, and the odds I was given. The distance between them is the only part I can work on. On a good week it is most of the page.

Harald