I exported everything. Three years of conversation with two different AI assistants, December 2022 through last week. 2,005 conversations. 9,640 turns that were mine.
The question I wanted answered was narrow. If I ever wanted to build something out of this archive, a model that wrote the way I write or decided the way I decide, was there anything in there worth learning from? Or was it nine thousand versions of "yes, do that"?
So I wrote a classifier to sort my turns into four buckets. Judgment calls, where I corrected something or set a constraint or ranked one option over another. Factual questions. Bare verdicts, which is a yes or a no with no reason attached. And everything else, mostly pasted code and fresh instructions.
It came back with 41.1 percent judgment calls. That felt high, and flattering, and I almost stopped there.
The instrument was measuring word count
Then I broke the number out by turn length, which is the check you run on any ratio before you quote it.
Turns under ten words: 9 percent judgment calls. Turns of 400 words or more: 95 percent. The curve climbs smoothly the whole way up, and it climbs almost identically on both corpora, which were written years apart about different things.
That identical shape is the tell. Two archives with nothing in common produced the same curve, which means the curve is not about the content. My classifier matched its markers anywhere in a turn, so a longer turn had more chances to trip one. I had not built a judgment detector. I had built a verbosity detector with a nicer name.
A second opinion made it worse. I ran a local model over 200 seeded turns as a check. On the label the entire question rests on, the two methods were not close: my classifier called 96 of those 200 a judgment call, the model called 11.
So the honest answer is that the real figure is somewhere between about 5 and 41 percent. That is a 36-point band. A band that wide is not a measurement, it is the absence of one dressed up in a decimal point.
The only instrument that actually collapses it is a hand-labelled sample. Three hundred turns, drawn across fourteen strata so the long turns where the disagreement lives are properly represented, read and labelled one at a time. I costed the reading: about seven and a half hours. Mine, not the machine's.
That is the joke at the center of this whole exercise. I built tooling to automate the counting of my own judgment, and the tooling's only honest output was an instruction to go read nine thousand words a bucket by hand. The drafting delegates. The parsing delegates. The counting delegates. The deciding what counts does not, and every hour I spent trying to make it delegate was an hour spent building a machine that measured length.
The number I wasn't arguing about
While I was busy distrusting the headline, a second column sat in the report doing nothing.
The classifier also flags turns where I changed my own mind. Not where I corrected the assistant. Where I conceded, or reversed, or said some version of you're right, I had that backwards.
Across 9,640 turns: 46.
That one is not in dispute the way the 41 percent is. Conceding has a much narrower vocabulary than judging, and it is short, so the length bias barely touches it. Forty-six is roughly one turn in two hundred and ten. In all of 2023, across 347 turns, the count is zero.
Sit with what a record like that is actually evidence of.
It is not evidence that I was right 9,594 times. Nobody is. It is evidence that across three years of daily conversation, the thing I was talking to almost never put me in a position where being wrong became visible.
That is not a defect in the tool, it is the tool working as designed. An assistant that is agreeable and fast and infinitely patient is genuinely useful for the work below the waterline. It is also, structurally, not a room where you get corrected. It reformulates your idea back to you slightly improved. It says great point and then builds the thing. It will disagree if you ask it to, which is not the same as being disagreed with, because you chose the moment and you set the terms.
Three years of that produces an archive with a very particular shape: a long, detailed, confident record of one person being right. Train anything on that and you get a machine that agrees with you, which is the failure I started this project specifically trying to avoid, sitting in the raw material the entire time.
What I'd carry out of this
Two things, and the second is the one that matters.
The small one: check whether your ratio moves with length before you quote it. Almost every number I have ever been burned by was a proxy for how much text there was.
The large one: your transcript is not a record of what you thought. It is a record of what the room allowed. A room with one agreeable participant produces forty-six reversals in three years, and a room with one person in it who will tell you the truth about your work produces forty-six in a good month.
I still don't know what fraction of those nine thousand turns carried real judgment. I'll know in about seven and a half hours of reading.
But I already know how often I was talked out of something, and it is not enough. That number was never the hard one to measure. It was just the one I wasn't looking at.